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Preface 


The  purpose  of  tliis  study  was  to  determine  if  software  reliability  models  can  be  applied  to 
the  Operational  Test  and  Evaluation  (OT&:E)  of  a  weapon  system  and,  if  this  was  the  case,  to 
implement  a  selected  model. 

An  extensive  review  of  current  literature  and  research  efforts  was  performed  to  identify  the 
candidate  models  for  evaluation  and  possible  implementation.  Models  were  evaluated  based  on 
predictive  validity,  capability,  quality  of  assumptions,  applicability  to  the  finite-time  environment, 
simplicity  of  design,  diversity  and  a()plicability  of  output,  and  capability  to  n.se  exist  ing  initial  data. 
From  thes<',  th('  Musa  Execution  Time  model  and  Musa-Okumoto  Logarithmic  Poisson  Execution 
Titne  model  were  selected  for  implementation.  The  implementation  was  testetl  using  data  supplied 
by  Headquarters  .Air  Force  Operational  Test  and  Evaluation  Center  (HQ  AFOTEC). 

I  would  like  to  thank  Capt  Jim  Cardow,  my  thesis  advisor,  for  his  guidance,  suggestions,  and 
especially  the  recurritig  question  “What  are  you  trying  to  do?”  I  would  also  like  to  thank  Lt  Col 
Lawlis  and  Maj  Howatt  for  reviewing  all  the  drafts  and  helping  me  to  remember  that  there  is  a 
forrest  and  not  just  one  tree.  I  also  thank  Dr  Moore  for  reviewing  my  derivations  and  providing 
statistical  insight. 

.A  special  thank  you  goes  to  my  wife  Lynn,  and  children  Devon.  Cheryl,  ami  Cara,  for  their 
patience,  understanding,  and  support  throughout  the  past  eighteen  months. 

Finally,  I  woidd  like  to  give  glory  to  the  Lord  Jesus  Christ  and  thank  Him  for  providing  me 
an  opportunity  to  learn  and  grow  during  the  A  FIT  experience. 
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\  AbsIracI 

..  '  Current  Air  Force  practice  is  to  perform  Operational  'lest  aiul  Evaluation  (OT&E)  for  each 
new  weapon  system.  In  support  of  tliis,  Headcfuarters  Air  F^orce  Operational  Test  and  Evaluation 
C'enter  (HQ  AFOTEC)  is  responsible  for  measuring  both  suitability  and  effectiveness.  While  suit¬ 
ability  is  adequately  measureil,  the  current  effort  only  aildresse.s  hardware  efi'ectiveness,  or  at  best, 
system  effectiveness.  Since  tools  and  metrics  are  in  place  for  software  suitability  assessments  related 
to  OTiCE  (for  example,  software  maintainability),  there  should  be  some  effective  way  of  mea.suring 
the  operational  effectiveness  of  software.  Currently.  HQ  AFOTEC/hf  b')  has  a  data  collection  tool 
for  collect ing  .soft wall'  failuri'  data  to  analyze  softwari'  maturity.  'I  his  thesis  propo.ses  that  the  bCb 
software  maturity  database  could  be  used  as  the  baselitie  for  a  .software  reliability  metric  that  would 

ttiap  to 

/ - 

W...  -  . 

This  study  does  tiot  predict  software  reliability,  nor  does  it  attempt  to  defitie  what  constitutes 
reliable  software.  Itistead,  this  study  evaluates  software  reliability  measurement  mapped  to  finite 
OTiAE  titne  fratnes  (i.e.-faihtres  per  flight  hour).  This  evaluatioti  is  cotiducted  for  several  software 
reliability  models,  with  two  candidate  niodel.s  chosen  based  on  the  following  criteria:  predictive 
validity;  capability;  ciitality  of  a.ssumptions;  applicability  to  the  finite-time  environment;  simplicity 
of  design;  diversity  and  applicability  of  outi>iif,  and  capability  to  use  existing  initial  data. 

Implementation  of  t  he  candidate  models  was  accomplished  for  an  office  computer  environment 
to  permit  ii.se  by  Od'A'E  test  teams  at  various  locations  'Jesting  was  performed  based  on  actual 
OTA'E  software  maturity  data. 


the  finite  time  0'l\CIv  environment. 
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A  STANDARDIZED  SOFTWARE  RELIABILITY  MEASUREMENT 

METHODOLOGY 


I.  Introduction 

The  overall  reliability  of  new  and  modified  weapon  systems  is  of  major  importance  to  the 
United  States  Air  Force  (USAF),  and  is  discussed  in  recent  standards  and  documents  that  address 
system  reliability  and  maintainability  [54:355].  Indeed,  many  authors  have  addressed  the  need 
for  software  reliability  evaluation,  both  in  journals  and  in  books  (reference  bibliography).  While 
hardware  reliability  can  be  virtually  guaranteed  at  delivery,  the  delivery  of  reliable  software  is  not 
as  predictable,  and  becomes  the  critical  factor  in  determining  system  reliability  [50:190].  This  thesis 
explores  the  possibility  of  implementing  software  reliability  measurement  as  part  of  the  Operational 
Test  and  Evaluation  (OT&E)  of  United  States  Air  Force  (USAF)  weapon  systems,  with  the  goal  of 
identifying  one  model  and  methodology  that  is  appropriate  for  use  in  the  Initial  OTA:E  (lOTfcE) 
phase.  As  Headquarters  Air  Force  Operational  Test  and  Evaluation  Center  (IIQ  AFOTEC)  is 
responsible  for  conducting  OTA;E  on  USAF  weapon  systems,  the  results  of  this  thesis,  as  well  as  a 
proposed  implementation  methodology,  are  then  submitted  to  HQ  AFOTEC  for  possible  inclusion 
in  their  software  evaluation  efforts. 

This  chapter  provide.s  the  background  of  software  development  and  testing,  and  identifies  the 
problem  with  .software  operational  testittg.  The  following  sections  will  define  hardware  and  software 
reliability,  establish  the  scope  of  this  thesis  effort,  identify  applicable  assumptions,  and  describe  the 
research  approach. 

y.J  Ba(  kgroinid 

Fhe  complexity  of  softwan'  in  future  systems  will  be  at  lea.st  an  order  of  magnitude  above  that 
of  current  .systems,  which  is  even  now  too  complex  for  one  individual  to  grasp  [13:3.5].  .Software 
cotnplexity  is  one  of  the  factors  affecting  the  overall  .software  cost  [30: 12-t5], [82:522].  Henry  and 
Kafura  state,  ■■reducing  cost  and  iticreasing  quality  are  compatible  goals  which  can  be  achieved 
when  the  complexity  of  the  software  structure  is  properly  controlled"  [37:510],  With  respect  to 
software  cost,  Myers  suggests,  “the  high  cost  of  software  is  largely  due  to  reliability  problems" 
[65:12],  Therefore,  a  software  cost  trend  might  be  an  indicator  of  the  underlying  complexity  of  the 
code  and  development  effort,  which  could  also  be  closely  relaterl  to  the  software's  reliability. 
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An  increasing  trend  in  software  cost  was  first  identified  in  a  study  by  tlie  Rand  Corporation 
for  the  Air  Force,  and  reported  in  [11]  and  [77]  as  a  substantial  increase  in  percentage  of  software 
cost  accompanied  by  a  corresponding  decrease  in  percentage  of  hardware  cost  (see  Figure  1.1) 
[11:1227], [77;11]. 


Figure  1.1,  Hardware  and  Software  Cost  Trends  (Reprinted  with  permission  from  IEEE) 

1.1.1  .■\ii  Vor(<  P( rsiHctivi.  Increased  software  cost  is  inclusive  of  software'  developme'iit 

and  support.  The  .Air  Force  has  doubled  its  spending  on  software  development  and  support  from 
S4  billion  in  1985  to  S8  billion  in  1989  [72:71].  A  recent  sltidy  of  37  Air  Force  Mission  Critical 
Computer  Hesource  (MCCK)  projects  evaluated  five  application  areas:  avionics:  communications: 
command,  control,  communication,  and  intelligence;  electronic  warfare:  and  radar  systems  [81:6]. 
The  study  stated  the  frequency  and  severity  of  change  in  software  size  contributes  to  cost  overruns, 
and  for  three  projects  the  actual  amount  of  software  developed  for  the  Air  Force  exceeded  the 
original  estimate  made  at  contract  award  by  100%  [81:7]. 

Corresponding  to  increasing  software  costs,  the  size  of  weapon  system  software  has  increased 
dramatically,  and  will  continue  to  increa.se.  This  increa.se  was  projected  by  Boehm  in  1976  and 
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reported  in  [77]  (see  Figure  1.2).  Current  estimates  of  the  amount  of  software  developed  for  DoD 
weapon  systems  have  verified  this  trend  (see  Figure  1.3)  [15:48]. 


Figure  1.2.  Projected  Cirowtii  in  Software  Memory  Rec|uirements  (Reprinted  with  permission 
from  McGraw-Hill  Book  Company) 


As  software  size  increases,  so  will  t  he  task  of  software  testing.  Lieutenant  Colonel  Shumskas, 
of  the  Office  of  the  Secretary  of  Defense,  responsible  for  .Air  Force  Test  and  Evaluation  (TvCE), 
suggested  the  following: 


It  is  possible  to  reduce  ac<|uisition  costs,  test  in  particular,  and  provide  software  intensive 
systems  with  increased  reliability  through  the  implementation  of  a  proposed  paradigm 
for  a  balanced  TAE  software  approach  utilizing  a  combination  of  statistical  process 
control  and  test  methodologies  [78:1-1]. 


.A  disciplined  test  methodology  could  then  help  recluce,  or  at  least  stabilize,  the  cost  of  software. 

With  respect  to  software  test  and  evaluation  of  a  weapon  system,  current  Air  Force  practice 
is  to  perform  Operational  Test  and  Evaluation  (O TArE)  under  the  direction  of  lleadtiuarters  Air 
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Figure  1.3.  Measured  Growth  in  Developed  Software, (Reprinted  with  permission  from  the  AFA) 

Force  Operational  Test  and  Evaluation  (.'enter  (HQ  AFOTEC)  for  each  new  weapon  system  fielded. 
This  effort  addrt*sses  system  performance  in  an  operational  scenario  (operational  effectiveness)  and 
system  availability  for  operational  use  (operational  suitability)  [2;l].  HQ  AFOTEC  has  tools  in 
place  for  evaluating  software  operational  suitability.  Unfortunately,  the  current  test  and  evalua¬ 
tion  effort  primarily  addresses  hardware  OTA:E  or.  at  b<>st.  system  OTAiE.  from  an  operational 
effect iveness  .standpoint. 

/.y.J  hidiistry  P(  r.sperttrf .  Industry  ha,s  also  addressed  the  need  for  software  reliability.  In 
one  of  the  first  papers  on  this  subject.  Mr.  Mulock  of  Lockheed  Missiles  Space  Company  wrote; 

The  Reliability  Engineer  should  consider  computer  programming  as  another  engineering 
tliscipline  that  is  analyzable  by  the  same  technique's  that  he  has  used  before  ...the 
computer  programmer  is  pushing  the  state  of  the  art  just  as  much  as  the  transistor 
designer  was  in  19.5.5  [59:497]. 
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Many  software  reliability  models  were  developed  during  the  subsequent  years,  and  recent  efforts 
have  defined  the  role  of  reliability  engineering  in  the  “typical  software  development  team”  [9;29l]. 
Industry  has  applied  several  of  the  software  reliability  models  to  projects  varying  from  remote 
terminal  firmware  (as  discussed  by  Musa  in  [61])  to  nuclear  power  plant  software  validation  [70].  In 
contrast,  there  has  been  little  use  of  software  reliability  measurement  for  military  weapon  systems 
[50]. 

As  recent  as  the  late  1980’s  software  for  major  military  command  and  control  systems  had 
proceeded  past  the  software  critical  design  reviews  without  any  assessment  of  software  reliability 
being  performed  [50:190].  In  contrast,  proposals  for  an  integrated  software  reliability  program  were 
being  presented  as  early  as  1976,  and  more  definitively  in  the  context  of  reliability  and  maintainabil¬ 
ity  in  198-4  [9.  65].  The  concept  of  software  reliability  has  been  in  investigation  for  over  15  years, 
and  several  different  organizations  such  as  Hewlett-Packard  Co..  ATicT,  and  the  .N'aval  Surface 
Weapons  Center  have  developed  and  used  software  reliability  tools  [29,  34,  61].  Goel  states; 

Software  reliability  is  a  useful  measure  in  planning  and  controlling  resources  during  the 

development  process  so  that  high  quality  software  can  be  developed  [34:1412]. 

Therefore,  the  use  of  software  reliability  assessment  could  be  one  of  the  disciplined  test  method¬ 
ologies  needed  to  curb  the  rising  cost  of  software. 

7.2  ProbUw 

M'hile  the  tools  and  metrics  are  in  place  for  .software  assessments  of  operational  suitability 
(for  example.  AFOTECP  800-2  \’ol  3,  Software  Matnimnubihly  Evaluation  Gvide  [22]),  there  is  no 
current  way  of  measuring  the  operational  effectiveness  of  software  in  order  to  perform  adequate 
software  OTA'L.  Reliability  is  considered  a  measure  of  operational  suitability;  however,  software 
reliability  is  also  one  possible  measure  of  the  software's  operational  effectiveiK'ss.  as  software  failures 
can  reflect  both  the  suitability  (|uestion  of  “will  it  be  available"  as  well  as  the  effectiveness  fiuestion 
of  “does  it  work"  [86:8-2], [2:8].  The  purpose  of  this  study  is  to  determine  if  software  reliability- 
models  can  be  applied  specifically  to  the  OTiA'E  of  Air  Force  weapon  systems  and,  if  this  is  the 
case,  to  propo.se  a  selected  model  implementation  within  a  standardized  methodology  for  use  by 
HQ  AFOTEC. 


1.3  Definitions 

Before  defining  software  reliability,  it  is  necessary  to  define  both  overall  reliability  and  hard¬ 
ware  reliability.  Reliability  is  defined  as 


the  (Juration  or  probability  of  failure-free  performance  under  stated  conditions  [20:8] 
and  also  as 


the  ability  of  an  item  to  perform  a  required  function  under  stated  conditions  for  a  stated 
period  of  lime  [5:29]. 

This  definition  is  primarily  based  on  the  system’s  hardware  attributes,  which  are  dis'^essed  below 
in  the  definition  of  hardware  reliability.  A  comparison  to  software  reliability  terms  will  then  follow. 

J.3.J  Hardware  Reliability  Terms.  Essentially,  hardware  reliability  is  defined  as  [20]: 

•  Mean  Time  to  Failure  (MTTF).  This  is  a  basic  measure  of  reliability  for  non-repairable  items, 
and  indicates  the  average  amount  of  time  until  the  failure  of  an  item. 

•  Mean  Time  to  Repair  (MTTR).  This  is  a  basic  measure  of  reliability  that  indicates  the  average 
amount  of  time  necessary  to  repair  an  item  once  it  has  failed. 

•  Mean  'l  ime  Between  Failures  (MTBF).  This  is  a  basic  measure  of  reliability  for  repairable 
items  and  is  definetl  as  a  combination  of  MTTF  and  MTTR  [20:7] 

M TBF  -  MTTF  -F  MTTR  (1.1) 


With  hardware,  these  attributes  can  apply  to  different  component,  subsystem,  and  system 
levels,  all  of  which  can  both  relate  to  each  other  and  have  discrete  values.  For  e.xample,  testing  of 
compoiK'iits  (such  as  integrated  circuits  (ICs))  can  be  performed  to  determine  the  MTTF.  MTTR, 
and  MTBF  values  for  each  1C.  This  value  can  then  be  included  in  the  calculation  of  subass«’mbly 
reliability,  which  can  have  cumulative  failure  rates  expressed  as  a  summation  of  the  components 

F,=YlfiN,  (1.2) 
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where  F,  is  the  subassembly  failure  rate  of  a  component  i.  given  its  failure  rate  /  and  a  certain 
number  of  like  components  N  [10;290].  This  sort  of  analysis  is  also  applicable  to  assemblies, 
subsystems,  and  finally  the  system  as  a  whole. 

1.3.2  Software  Reliability  Terms.  Software  does  not  permit  the  composition  and  decompo¬ 
sition  analysis  that  is  possible  with  hardware.  Using  a  software  instruction  (i.e. — x:=x+l)  as  an 
analogy  to  the  component,  we  find  that  an  instruction  functions  perfectly  100%  of  the  time  with¬ 
out  problems  on  its  own;  however,  by  combining  instructions  it  is  possible  to  develop  subroutines 
that  have  failures  (usually  due  to  subroutine  interactions)  [10:291].  Therefore,  software  subroutine 
reliability  can  not  necessarily  be  derived  from  the  corresponding  instruction  reliabilities.  Hard¬ 
ware  assemblies  are  then  created  from  subassemblies,  just  as  software  modules  (discrete  program 
units  that  are  "a  logically  separable  part  of  a  program")  are  made  up  of  subroutines  (a  routine, 
or  “computer  program  segment  that  performs  a  specific  task."  that  can  be  included  in  other  rou¬ 
tines)  [5:24,30,34].  While  hardware  assembly  reliability  can  be  determined  from  subassemblies,  a 
guarantee  does  not  exist  that  software  module  reliability  is  ba.sed  on  the  corresponding  subroutine 
reliabilities.  This  correlation  becomes  even  smaller  as  the  software  modules  are  linked  together 
into  subsystems,  and  finally  systems.  Thus,  the  standard  terminology  used  for  hardware  reliability, 
including  equations  1.1  and  1.2,  is  not  applicable. 

Instead,  a  different  view  of  software  must  be  taken.  This  is  based  on  the  design  of  the 
software,  and  not  the  physical  implementation  usually  measured  by  hardware  reliability  [64:7]. 
Several  sludif's  have  attempted  to  combitte  hardware  and  software  theory  into  a  system  reliability 
perspective,  citing  software  reliability  models  maturing  to  a  point  common  with  their  hardware 
counterparts  [31.  42].  By  viewing  software  reliability  as  an  integrated  aspect  of  system  reliability 
frotn  a  design  viewpoint,  it  is  possible  to  implement  software  reliability  theory  in  a  compatible  way 
with  hardware  reliability  theory  [64:7].  Ba.s«'d  on  this,  software  re liahihiy  \s  defined  as: 


The  jirohability  (hat  software  will  not  cause  the  failure  of  a  system  for  a  specifieel  time 
under  specified  conditions  [5:32], [40: 14]. 

The  prohahility  that  a  software  system  will  operate  without  a  failure  for  a  specified 
(mission)  time  [19:9-1]. 

The  prohahility  of  failure-free  operation  of  a  computer  program  for  a  specified  time  in 
a  specified  environment  [64:15]. 


A  failurt  is  defined  (with  respect  to  software)  a.s: 


An  event  in  which  a  system  or  system  component  does  not  perform  a  required  func¬ 
tion  within  specified  limits.  A  failure  may  be  produced  when  a  fault  is  encountered 
[5: 19],  [40;  14] 


wliere  a  fault  is: 


A  manifestation  of  an  error  in  software.  A  fault,  if  encountered  may  cause  a  failure. 
[5:19], [40:14]. 

Addit  ionally  a  new  concept,  the  failure  rate,  is  defined  as; 


The  ratio  of  the  number  of  failures  of  a  given  category  or  severity  to  given  period  of 
time  [5:19], 

These  definitions  permit  the  use  of  reliability  constructs  similar  to  those  used  with  hardware.  One 
e.xample  of  this  is  the  failure  intensity  representation  presented  by  Musa,  et  al.  [64:11.528-529] 
which  indicates  the  number  of  failures  per  unit  time  expressed  by  the  function 

A(r)  =  Ao<'^'  (1.3) 

where  A(r)  is  (he  current  measured  software  reliability  based  on  time  (r),  initial  failure  intensity 
(Ao)  and  total  estimated  failures  (I'o).  The  value  A(r)  is  the  current  failure  intensity,  and  indicates 
the  ratio  of  failures  to  operational  time.  .\  number  such  as  this  ratio  could  then  be  used  to  provide 
the  operational  assessment  of  computer  software. 


/.,/  Scope 

Currently.  HQ  .AFOTEC/LC15  has  a  data  collection  tool  for  collecting  software  failure  data 
to  analyze  and  determine  the  software  maturity.  The  data  are  identified  by  standard  data  item 
descriptions,  and  can  be  provided  either  as  part  of  (he  initial  contract  or  (hrongh  a  letter  to  the 
system  program  office  (SPO)  rec^uesting  the  data  to  support  .software  test  (see  Table  1.1)  [23:4],  The 
existing  software  maturity  databases  are  implemented  on  a  formal  for  office  personal  computers 
(PC's),  and  were  enhanced  to  permit  an  operational  effectiveness  a.ssessment  of  the  software  based 
on  the  candidate  software  reliability  models.  Compatibility  with  the  maturity  database  and  data 
collection  tool  required  the  software  reliability  model  implementation  to  be  in  the  same  program 
development  environment.  This,  in  turn,  allows  for  future  software  maturity  databases  to  be  used 
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Table  1.1.  Software  Maturity  Data 


Description 

"Variable  Name 

Format 

Software  Problem  Number 

PROB.NUM 

Character  10 

Software  Configuration  Item 

CPCl 

Character  10 

Severity  of  Problem 

SEV.CODE 

Character  1 

Date  Problem  Discovered 

DATE 

Date 

Date  Problem  Fixed 

DATE 

Date 

Description  of  Problem 

TITLE 

Character  42 

Total  Operating  Time  (minutes) 

TOT.TIME 

Character  10 

Test  Identification  Number 

TEST.1D 

Character  10 

Date  Test  Planned 

TESTPLAN 

Date 

Date  Test  Completed 

TESTCOMP 

Date 

as  a  baseline  for  software  reliability  assessment  during  the  finite  time  OTA;E  period.  The  use 
of  initial  maturity  data  (collected  prior  to  OT&E)  to  lay  the  foundation  for  software  reliability 
assessment  during  OTi:E  is  supported  by  Ferens,  who  states  that  software  reliability  models  “are 
only  useful  after  testing  begins’’  [30:11-4]. 

A  previous  effort  by  Westgate  [92]  attempted  to  validate  a  predtclivt  model  of  software  relia¬ 
bility.  While  prediction  has  its  necessary  place  in  software  evaluation,  this  study  does  not  attempt 
to  predict  software  reliability,  nor  attempt  to  identify  what  number  correlates  to  reliable  soft¬ 
ware.  Instead,  this  study  evaluates  software  reliability  nifasvnvteni  models — models  that  indicate 
the  current  software  reliability  without  making  any  determination  of  the  overall  quality  of  the 
software — mapped  to  finite  OTi;E  time  frames  (i.e. — failures  per  flight  hour).  This  study  also 
attempts  to  determine  the  type  of  assessment  such  models  could  provide.  The  determination  of  “is 
the  software  reliable  enough"  and  “how  much  more  testing  is  needed”  can  then  be  decided  by  the 
decision  makers  based  on  a  reporting  of  “where  is  the  software  right  now." 


}.i  Assiiniphonn 

This  study  presents  no  new  models  for  evaluating  software  reliability.  The  e.xisting  models 
wore  assumed  to  be  valid  with  respect  to  the  entire  life-cycle  of  a  software  development  effort.  The 
main  focus  is  on  the  specific  mapping  to  a  finite  OTAjE  time  frame. 

Several  different  categories  of  OTijE  e.xist.  an<l  it  is  assumed  that  only  the  Initial  OTit;E 
(lOTijE)  period  will  be  used  for  time  constraints  [2].  As  lOTAiE  supports  procurement  decisions, 
and  Follow-on  OT&E  (FOTicE)  begins  after  a  weapon  system  enters  production  [2:1-2],  the  IOT&:E 
timeframe  is  better  suited  to  pre-production  software  assessment. 
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1.6  Approach 


The  first  step  was  to  conduct  a  literature  review  of  the  models  available  for  software  relia¬ 
bility  evaluation.  Identification  and  classification  of  these  was  performed  based  on  their  individual 
characteristics  and  focus.  From  these,  models  were  selected  for  possible  mapping  into  the  lOT^cE 
time  frame  based  on  the  following  criteria;  predictive  validity  (of  the  model’s  parameters,  not  the 
reliability  itself);  capability;  quality  of  assumptions;  applicability  to  the  finite-time  environment; 
simplicity  of  design;  diversity  and  applicability  of  output;  and  capability  to  use  existing  initial  data 
[39],[64;384-387]. 

Implementation  of  two  candidate  models  was  attempted  to  further  validate  their  usefulness 
for  evaluating  software  operational  effectiveness.  The  implementation  was  conducted  in  accordance 
with  the  software  engineering  di.scipline  approach,  and  encompassed  a  relational  databa.se  design 
to  permit  data  persistence.  The  design  environment  for  program  development  was  the  Clipper 
programming  environment.  The  target  system  is  any  MS-DOS  office  computer  environment  to 
allow  use  by  IOTi;E  test  teams  at  various  locations. 

1.7  Thesis  Organization 

A  review  of  available  software  reliability  models  is  reported  in  Chapter  2.  Chapter  3  describes 
the  evaluation  of  the  models  and  selection  of  the  candidate  model.  Implementation  of  the  candidate 
model  is  documented  in  Chapter  4.  with  the  findings  and  results  given  in  Chapter  5.  Finally, 
conclusions  and  recomnu'ndat ions  are  presented  in  Chapter  (5. 
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II.  Literature  Review 


The  software  engineering  discipline  itself  is  concerned  with  “the  systematic  application  of 
methods,  tools  and  technical  concepts  to  create  complex,  software-intensive  systems  that  meet 
technical,  economic  and  social  objectives”  [32:32].  One  such  technical  concept  is  software  quality, 
which  is  defined  as  “the  totality  of  features  and  characteristics  of  a  software  product  that  bear  on 
its  ability  to  satisfy  given  needs”  [5:32].  Indeed,  software  reliability  has  been  identified  as  one  of 
several  software  quality  factors  that  affect  the  software  life-cycle  and  its  associated  cost  [30,  90,  91]. 

There  are  several  suggested  frameworks  for  identifying  the  software  quality  factors  [52,  87,  91], 
Tindell  [87]  investigated  complexity  for  the  maintenance  of  JOV'1.4L  J73  software,  and  identified  a 
software  quality  framework  which  included  complexity  and  reliability  (see  Figure  2.1).  Johnson  also 
evaluated  complexity  and  its  metrics,  in  this  case  for  use  in  the  .■VFOTECP  800-2  Vol  3.  Software 
Mamiainabthiy  Evaluation  Guide  [44].  These  works  focused  on  the  operational  suitability  assess¬ 
ments  for  software  OTfcE.  In  comparison,  VVestgate  addressed  the  software  quality  of  reliability  in 
evaluating  a  software  reliability  model  for  software  OTi:E  that  uses  calendar  time  as  a  basis  [92]. 
To  compliment  these  efforts,  this  thesis  also  explores  software  quality;  however,  the  focus  is  on 
the  operational  effectiveness  of  the  software  based  on  reliability  as  derived  from  test,  or  execution, 
time.  This  chapter  identifies  endeavors  in  the  literature  to  address  software  reliability. 

~.I  Software  Reliability  .Model  Classifications 

Many  attempts  havp  been  made  to  define  the  concept  of  software  reliability  and  determine 
some  form  of  software  reliability  assessment  model  [10,  17,  28.  40,  43,  55,  60.  62,  65.  68,  69,  73,  74, 
7\j,  86,  88,  96],  Such  efforts  have  provided  excellent  insight  into  specific  areas  of  soft  ware  reliability 
evaluation.  However,  software  reliability  models  currently  available  are  not  considered  “universally 
appropriate  ’  across  all  application  domains  and  system  usages,  ami  Sommerville  suggests  that  “it 
may  be  appropriate  to  use  different  reliability  metrics  for  dilfereiit  parts  of  the  system"  [82:596], 

Paralleling  the  efforts  of  model  definition  are  consolidations  of  software  relial^ility  definitions 
and  models  into  a  single  compendium  or  reference  handbook  [19,  27,  34,  41,  56,  57,  64,  80,  83,  85], 
Such  attempts  take  several  software  reliability  models  and  group  them  by  some  cla,ssification, 
allowing  the  software  engineer  to  select  the  appropriate  method  for  a  specific  application.  The 
following  are  major  efforts  to  classify  ,software  reliability  models. 

2.1.1  IEEE  Classification.  The  Institute  of  Electrical  ami  Electronics  Engineers  (IEEE) 
classifies  software  reliability  models  in  terms  of  product  measures  and  process  measures  (see  Table 
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2.1)  [41;25-27].  Process  measures  provide  input  for  the  processes  of  both  development  and  sup¬ 
port  management,  and  include;  using  management  control  measures  for  fault  removal  cost  trends; 
coverage  to  ensure  completeness  of  activities  throughout  the  software  life-cycle;  and  technical  and 
cost  evaluations  for  software  delivery  decisions  [41:25].  Indicators,  such  as  testing  sufficiency,  are 
similar  to  those  in  the  Air  Force  Systems  Command  Pamphlet  (AFSCP)  800-14,  Software  Quality 
Indicators  [25].  A  study  by  Lipow  [53]  identifies  one  approach  that  uses  a  form  of  residual  fault 
count  and  error  distribution  measures  [41]. 

Product  measures,  on  the  other  hand,  focus  on  the  developed  software  objects  and  encompa.ss 
many  different  metrics  such  as  fault  density,  failure  rate,  and  mean-time-to-failure  [41:26-27].  These 
measures  are  applicable  to  both  software  reliability  prediction  and  measurement  models.  With 
respect  to  software  reliability  prediction  models,  Wilson  and  Shen  state: 

No  growth  model  has  demonstrated  that  it  can  be  used  with  a  high  degree  of  confidence 
to  predict  operational  reliability  from  data  generated  in  the  debugging  phase  in  a  general 
setting  [93:5]. 

In  contrast,  the  focus  of  OT&E  is  to  field  test  and  evaluate  weapon  systems  to  determine 
effectiveness  and  suitability  [2].  AFR  800-18,  Air  Force  Reliability  and  Minntainabiliiy  Policy, 
requires  implementing  “reliability  qualification  and  acceptance  testing,  ...[which]  will  be  tailored 
for  effectiveness  and  efficiency  in  terms  of  the  management  information  they  provide"  [24:3],  .Most 
software  reliability  models  re<]uire  data  for  calibration;  however,  it  is  not  possible  to  directly  measure 
software  reliability  during  the  design  and  coding  stages  where  such  calibration  data  floes  not  exist 
[30,  63],  Therefore,  specific  product  measures  for  OTAcE  assessment  of  software  reliability  should 
focus  on  ineasvreinent  of:  errors,  faults  and  failures;  mean-time-to-failure  and  failure  rates;  and 
remaining  product  faults  [41:2t5]. 

“2.1.2  A'.S'll'C  Cla.ssification.  The  .Naval  Surface  Weapons  fenter  (NSWC)  'I'echnical  Report 
(TR-82-171)  classifies  software  reliability  models  into  three  categories:  error  seetliTig/tagging  mofl- 
els;  data  domain  approach  model.;;  and  lime  floiiiain  approach  models  [27],  Error  seeding/tagging 
nioflels  are  “built  on  firm  statistical  ground"  [65:336].  The  original  work  by  .Mills  (as  described  by 
Myers  in  [65])  developed  a  soft  ware  reliability  model  requiring  software  engineering  personnel  to 
place,  or  "seed"  errors  intentionally  in  the  computer  software.  The  errors  are  seeded  raiulomly,  with 
the  assumption  that  an  equal  probability  exi.sts  of  finding  either  seeded  or  original  errors  during 
testing.  Since  the  number  of  seeded  errors  is  known  a  priori,  the  ratio  of  the  number  of  found 
seeded  errors  divided  by  total  seeded  errors  would  he  equal  to  the  ratio  of  the  number  of  found 
original  errors  divided  by  total  original  errors  [65], 


Table  2.1.  IEEE  Software  Reliability  Model  Classification 


Product  Measures 

Process  Measures 

Errors,  Faults,  Failures 

Management  Control 

Mean-Time- To- Failure, 
Failure  Rate 

Coverage 

Reliability  Growth  and 
Projection 

Remaining  Product  Faults 

Risk,  Benefit, 

Cost  Evaluation 

Completeness  and  Consistency 

Complexity 

Data  domain  approach  models  are  similar  to  error  seeding/tagging  models  in  that  they  esti¬ 
mate  a  program's  current  reliability  from  a  ratio.  In  this  case,  the  ratio  is  the  number  of  successful 
test  runs  completed  divided  by  total  number  of  test  runs  attempted  [27:3-1].  This  ratio  a.ssumes 
that  there  is  an  equal  probability  of  either  failure  or  success  for  each  test  run  [19:9-21].  The  test 
inputs  are  chosen  based  on  probability  distributions  estimated  for  operational  u.se.  and  the  .success 
of  a  test  run  is  defined  with  respect  to  these  inputs  [27:3-1]. 

Time  domain  approach  models  mode)  the  error  generation  process  based  on  errors  and  time. 
The  relationship  between  the  two  is  based  on  either  error  occurrence  times  and  the  calculated  times 
between  error  occurrences,  or  the  number  of  errors  that  occur  during  a  specified  lime  period  [27:-l- 
1].  Several  of  these  models  are  similar  to  hardware  reliability  models,  and  use  major  a.ssumptions 
conc<’rning  the  probability  distribution  of  software  failures  [t)5:330]. 

With  in  the  lime  domain  approach  models,  there  is  a  further  distinction  based  on  the  spe¬ 
cific  mathematical  method  u.sed.  The  NSWC  report  identifies  three  subcategories  of  time  domain 
approach  models:  classical  software  models;  Bayesian  models:  and  .Markov  models  [27).  Both  the 
cla.ssical  software  models  and  Fiayesian  models  treat  software  reliability  as  a  function  of  continuous 
events.  The  classical  software  models  use  probabilities  deriverl  from  software  failure  frequency  anal¬ 
ysis  and  software  hazard  (or  failure)  rates,  the  Bayesian  models  use  a  more  subjective  viewpoint  in 
counting  errors  [38]. [27:100-101]. 

In  contrast  to  these,  the  Markov  models  view  .software  reliability  as  a  .series  of  discrete  events 
[27:116].  Markov  models  treat  each  software  failure  event  as  a  separate  occurrence,  such  that  an 
event  at  time  /-t-l  is  not  based  on  the  reliability  history  previous  to  time  f  (38:5-l.'j]. 

2.J.S  HA  DC  dasstficaiion.  Home  .Air  Development  Center  (RADC)  drew  upon  an  earlier 
work  of  Coel  to  identify  four  classes  of  software  reliability  models:  fault  seeding  models:  input 


domain  models;  times  between  failure  models;  and  failure  count  models  [56:3-35].  The  categories 
of  fault  seeding  models  and  input  domain  models  are  identical  to  the  NSWC  categories  of  error 
seeding/tagging  models  and  data  domain  approach  models,  respectively  [27,  56], 

The  times  between  failure  models  category  is  similar  to  the  Bayesian  subclass  of  the  NSWC 
time  domain  approach  models,  while  the  failure  count  models  are  identical  to  the  claissical  subclass 
of  NSWC  time  domain  approach  models  [27,  56].  Goel  uses  this  same  classification  again  in  a 
later  article  addressing  the  assumptions,  limitations,  and  applications  of  various  software  reliability 
models  [34]. 

“2.1.4  M1L-HDBK-33S-L\  Classificalton.  MlL-HDBK-338-1  A  defines  a  higher  level  of  ab¬ 
straction.  classifying  software  reliability  models  into  two  general  categories:  non-failure  rate  based 
models  and  failure  rate  based  models  [19]. 

2. 1.4.]  Noil-Failure  Rate  Based  Models.  The  term  non-failure  rate  implies  that  the 
software  reliability  model  is  independetit  of  the  software’s  failure  rate  [19,  65].  The  two  basic  types 
of  non-failure  rate  based  models  are  combinatorial  and  input  domain  [19].  Combinatorial  models 
derive  their  name  from  the  mathematical  formula  of  ratios  of  identified  faults  to  e.xpected  faults 
[19].  The  combinatorial  models  include  both  error  seeding  and  binomial  models  [19]. 

While  the  input  dotiiain  category  is  identical  to  the  RADC  input  domain  category  and  the 
error  .seeding  combinatorial  model  category  is  identical  to  the  RADC  fault  seeding  category,  there  is 
no  corresponding  category  for  the  bitioitiial  tnodels  [19.  56].  The  binomial  models  use  cotnbinatoric 
mathematics  to  calculate  reliability  probability  from  the  number  of  errors  encountered,  the  number 
of  attempted  program  test  runs,  and  the  probability  of  finding  errors  on  any  given  program  test 
rtm  [19:9-21].  While  such  a  method  is  appealing  based  on  its  simplicity,  it  is  more  a  predictor  thatt 
a  measure  of  software  reliability,  and  will  not  be  considered  further. 

2. 1.4.2  Failure  Rale  Based  .Meidels.  In  contrast,  failtire  rate  based  models  are  con¬ 
cerned  with  the  number  of  software  failures  and  the  freqtiency  at  which  they  are  exiterienced 
during  a  period  of  time  [65:330-331].  MIL-HDBK-338-1.A  identifies  two  categories  of  failure  rate 
based  models:  classical,  and  Bayesian  [19].  These  categories  map  directly  to  the  subcla.sses  of  clas¬ 
sical  and  Bayesian  of  the  NSWC  time  domain  approach  models,  as  well  as  the  R.ADC  categori«'s  of 
failure  count  models  and  times  between  failure  models,  respectively  [19,  27.  56], 

2.1.5  Musa  and  Okumoio  Classificeilion.  In  the  book  .Software  Reliability:  Measiiitvienl. 
Prediction.  Applicalion.  Musa  et  al.  give  a  different  classification  scheme  first  presenti'<l  by  Musa 


and  Okumoto  in  1983.  Model  classification  is  based  on  five  attributes:  time  domain  (calendar  time 
or  e.xecution  time);  category  (either  a  finite  or  infinite  number  of  failures  experienced  in  infinite 
time);  the  distribution  type;  class  (only  if  the  model  is  in  the  finite  failure  category);  and  family 
(only  if  the  model  is  in  the  infinite  failure  category)  [64;250-251].  The  table  from  Musa  et  al.  is 
shown  in  Figure  2.2. 

Musa  el  al.  discuss  models  with  respect  to  both  time  domains;  however,  execution  time  better 
“characterizes  the  failure-inducing  stress  placed  on  software”  [64:31].  Therefore,  only  the  execution 
time  based  models  will  be  discussed.  Within  the  Musa  and  Okumoto  classification,  a  model  is  first 
identified  as  either  a  finite  or  infinite  failure  model  depending  on  whether  the  model  assumes  a  finite 
or  infinite  number  of  failures  will  be  reached  at  time  /  =  oc'  [62:23.5].  Next,  the  failure  quantity 
distribution  for  failure  experienced  at  time  t  is  identified  [62:235]. [64:250].  Three  distributions 
have  been  identified  for  the  finite  failure  category,  while  four  have  been  identified  for  the  infinite 
failure  category  [62:235], [64:250, 251].  Against  these,  the  failure  intensity  form  is  cross-referenced, 
using  time  as  a  basis  for  the  class  (finite  failure  category)  and  expected  number  of  failures  as  a 
basis  for  the  family  (infinite  failure  category)  [62:235], [64:250].  This  type  of  analysis  identifies  the 
relalionsliips  betweeti  models  within  both  of  the  times  between  failure  and  failure  count  categories 
[56,  64]. 

2.1.6  Overall  Model  Classificatton  Schema.  A  comparison  of  the  .MlL-llDBK-338.4,  NS\VC‘, 
and  R.ADC  software  reliability  model  classifications  is  shown  in  Table  2.2.  From  this,  and  Figure 
2.1.  the  concept  map  in  Figure  2.3  was  derived.  This  softwar<‘  reliability  concept  map  reflects  the 
overall  cla.ssification  as  identified  in  the  previous  sections.  The  initial  division  of  software  reliability 
into  process  measures  and  product  measures  is  based  on  the  IEEE  cla.ssification.  While  the  process 
measures  are  very  important  to  the  management  of  the  overall  software  life-cycle,  the  OTfcE 
effort  recjiiires  an  ap|)roach  that  evaluates  the  .software,  and  not  the  management  process  [2].  The 
additional  level  of  abstraction  defined  by  .MlL-nDBI\-,338-l.\  (identifying  failure  and  non-failure 
rate)  would  be  plan'd  between  the  IEEE  product  measures  and  the  lower  categories,  and  isomittetl 
for  clarity.  Snbse(|uenl  divisions  of  the  product  measurement  into  software  reliability  models  are 
ba.sed  on  the  categories  derived  from  the  N'SWC.  RADC,  aiul  (.loel  classifications,  and  are  identified 
as  the  model  categories  of  fault  seeding,  input  domain,  limes  between  failures,  and  failure  count. 
While  the  Musa  el  al.  software  reliability  model  cla.ssifiralion  differs  from  this  more  traditional 
hierarchy,  it  does  prove  useful  for  relating  models  to  each  other  within  appropriate  classifications. 
This  relationship  is  important  for  identifying  initial  models  for  evaluation. 
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2.^  Soflwart  Reliability  Model  Descriptions 

The  following  major  models  were  identified  for  evaluation  from  the  four  model  categories; 

•  Fault  Seeding:  Mill’s  Hypcrgeometric  model  [65] 

•  Input  Domain:  Ramamoorlhy-Bastani  model  [69] 

•  Times  Between  Failures:  Jelinski-Moranda  model  [43],  Littlewood-V'errall  moilel  [55],  Schick- 
Wolverlon  model  [73], 

•  Failure  Count :  Goel-Okumoto  Nonhomogeneous  Poisson  Process  model  [35],  Musa  Execution 
Time  model  [60],  Musa-Okumoto  Logarithmic  Poisson  Execution  Time  model  [62],  Shooman 
Exponential  model  [76],  ^'amada-Ohba-Osaki  Power  model  [96] 

Other  software  reliability  models  (especially  times  between  failures  and  failure  count  models)  are 
similar  to  these,  being  either  more  generalized  or  refined  for  specific  applications  [27,  34,  56,  64. 
86].  The  focus  on  evaluating  older  models  is  permissible,  as  there  have  been  no  significantly  new 
software  reliability  models  developed  in  the  la.st  eight  years  [79].  Each  model  s  assumptions  follow 
its  description.  Goel  states  the  following  concerning  software  reliability  model  assumptions 

. .  .as  a  totality,  they  provide  an  msight  into  the  kind  of  limitations  imposed  by  them 
on  the  use  of  the  software  reliability  models  . .  .The  ultimate  decision  about  the  appro- 
priateiu'ss  of  the  underlying  assumptions  and  the  applicability  of  the  models  will  have 
to  be  made  by  the  user  of  a  model  [34:1417], 

Therefore,  the  a.ssumplions  will  be  identified  in  this  chapter,  and  their  ap|)licability  will  be  assessed 
in  the  following  cliapter. 

2.2.1  Fault  Seeding  Models.  Goel  identifi«’s  the  two  major  assumptions  necessary  to  u.se 
fault  st'eding  models  [34:1419]: 

•  Faults  are  seedetl  randomly  throughout  the  program. 

•  Innate  faults  have  the  same  probability  as  the  seeded  faults  of  being  discovered  during  test. 
.\  discussion  with  respect  to  the  major  model  follows. 


Mill's  Hypergeotntinc  Model. 


Equation.  The  genera!  equation  for  this  model  is  given  in  Myers  [65] 

A’  =  sn/v  (2.1) 


where  N  is  the  inaximiini  likelihood  estimate  of  total  number  of  innate  errors,  n  is  the  number 
of  detected  innate  errors,  s  is  the  total  number  of  seeded  errors,  and  u  is  the  number  of  detected 
seeded  errors  [65:336-337],  The  confidence  calculation  C  is  also  given  in  Myers  [65] 


C  = 


1  if  n  >  A- 


f+t+i 


if  n  <  I: 


where  k  is  an  upper  bound  assumption  of  the  number  of  innate  errors  in  the  program  [65:337], 

Assuinplions.  Although  the  error  detection  probabilities  are  unknown,  the 
Mill’s  model  assumes  both  the  innate  and  seeded  errors  have  the  same  detection  probability 
[65:337],  Random  error  seeding  throughout  the  program  is  another  important  assumption;  how¬ 
ever.  seeding  errors  that  have  the  same  probability  of  detection  as  innate  errors  is  a  major  problem 
[6: 12], [3-1: 14 19], 

2. ‘2. ‘3  Input  Domain  .Models.  The  major  assutnptioiis  necessary  for  input  domain  models 
are  summarized  by  Goel  as  [34:1419]: 

•  Testing  performeil  is  random. 

•  The  dislril>ution  is  known  a  prion  of  the  input  profile  for  lest. 

•  Input  domain  e<|uivalence  clas.ses  can  be  determined. 

A  discu.ssion  with  respect  to  the  major  model  follows. 


Ramamoorth  y-  Bus  lam  Mode  I. 

Equation.  The  Ramamoorthy-Bastani  motlel  is  defined  as  [69] 


n  —  1 


P{E,\n}  =  r-^^'  n 


z  =  i 


1  + 


(2.2) 
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based  on  a  program’s  continuous  equivalence  class  specified  by  Ei  =  [a,  a  +  r],  with  n  test  cases 
each  having  successive  distances  for  j  =  1,  •  ■  • ,  n  —  1  [69:366].  Here.  X  is  the  inverse  of  the  mean 
length  of  intervals  for  Ei,  and  V  is  a  determination  of  the  number  of  errors  [69:366].  The  product 
AV  is  related  to  both  the  number  N  of  elements  in  and  degree  D  of  an  equivalence  class  [69] 


Assumpitons.  The  Ramamoorthy-Bastani  model  assumes  the  input  can  be 
divided  into  equivalence  classes,  and  then  requires  an  assumption  of  the  equivalence  class  distribu¬ 
tion;  however,  the  determination  of  the  equivalence  clas.ses  is  very  costly  [3'i:l-419]. [69:367].  It  also 
allows  the  use  of  any  test  case  selection  strategy,  and  does  not  assume  random  sampling  for  test 
inputs  [69:367]. 

2.^.3  Times  Between  Failures  Models.  Goel  discusses  several  assumptions  common  to  the 
times  between  failures  models  [3*1:1417-1419]: 

•  Faults  are  independent  and  have  the  same  probability  of  e.\posure. 

•  Perfect  debugging  is  done  immediately  after  the  occurrence  of  a  fault. 

•  Successive  times  between  failure  occurrences  are  independent  of  each  other. 

•  The  software  system  failure  rate  decrea.ses  as  testing  proceeds. 

.•\  discussion  with  respect  to  the  major  nrodels  follows. 

J<  Iniski-Moianda  Model. 

Equation.  The  Jeliiiski-Moranda  motlel  defines  the  probability  of  a  time 
interval  between  the  i  —  1  and  ith  consecutive  errors  as  [1.3] 

P{jLi)  =  <i[.V  -  (#  -  i)]f-^('  -"-"P-  (2.3) 

where  A’  is  the  initial  error  content  and  O  is  a  proportionality  constant  [43:473].  The  hazard 
function  c(/,)  is  defined  by  the  software  failure  rate  d>[A'  —  (»  —  1)]  [43:47.3].  Musa  et  al.  takes  this 
a  step  further,  and  derives  the  failure  intensity  function  with  respect  to  time  (A(<))  based  on  the 
constant  hazard  rate  di  [64] 

A(/)  =  A’c.r-»' 
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Assunipilons.  A  major  assumption  of  this  and  other  times  between  failure 
models  is  based  on  perfect  debugging,  the  act  of  fault  correction  without  introducing  new  faults 
[34;1418].  Another  assumption  shared  by  models  in  this  category  is  the  independence  of  successive 
failure  times  from  each  other  [34:1417].  The  model  also  2tssumes  the  failure  rate  between  errors  is 
uniform  [43:473],  This  notion  of  a  constant  arrival  rate  for  errors  has  been  cited  as  a  drawback 
[73:105],  Also,  testing  time  periods  which  are  of  equal  length  are  assumed  to  represent  the  same 
thoroughness  of  testing  [43:477],  Musa  et  al.  categorize  the  Jelinski-Moranda  model  as  a  finite  fail¬ 
ure  e.xponential  class  model,  which  assumes  that  at  infinite  time  the  number  of  failures  experienced 
is  finite  [64:278-280]. 

LtHlewood-  V’erralt  Model. 

Eqnahon.  The  equation  for  the  Littlewood-V’errall  model  is  [55] 


<l(l  I  ».o)  =  < 


r(o) 


0 


/  >  O.u-  >  O.o  >  0 

/  <  0 


(2,4) 


where  the  hazard  rate  /  is  expressed  as  the  probability  density  function  ij(l  \  >,o),  is  the  growth 
function  for  the  gamma  distribution,  and  o  is  the  shape  parameter  for  the  gamma  distribution 
[55:110],  The  probability  density  function  for  time  of  next  failure  1^  after  repair  of  the  previous 
failure  given  the  failure  rate  A  is  [55:1 10] 


fit  I  A)  =  ^ 


Af--'' 

0 


<  >  0,  A  >  0 
I  <  0 


.Musa  et  al.  <lefine  the  failure  intensity  function  with  respect  to  time  (A(t))  based  on  these  probability 
density  functions  as  [64] 

A(f)  =  ——2= 

with  /iu  and  .^i  being  model  parameters  of  the  reliability  growth  function  v  [64:294-296], 

.Assuwptwvs.  A  major  assumption  of  this  and  other  times  between  fail¬ 
ure  models  is  based  on  perfect  debugging,  where  fault  correction  occurs  before  finding  the  next 
fault  without  introducing  new  faults  [55:109],  The  independence  and  randomness  of  successive 
failure  times  are  other  assumptions  shared  by  models  in  this  category  [55:109],  .As  it  takt^s  a 


2-12 


Bayesian  approach,  this  model  assumes  “subjective  attitudes  to  the  system  under  consideration, 
thus  probability'  means  ‘personal  probability’  or  ‘degree  of  belief’  ”  [55:110],  Musa  et  al.  classify 
the  Littlewood- Verrall  model  as  a  member  of  both  infinite  failure  inverse  linear  and  inverse  poly¬ 
nomial  families,  which  assumes  that  at  infinite  time  the  number  of  failures  experienced  is  infinite 
[64:293-296], 

Schtck-  Wolverton  Model. 

Equaiion.  The  original  Schick- Wolverton  model  (as  described  by  Schick 
and  Wolverton  in  [73])  is  given  by: 


/?(<,)  =  (2.5) 

with  the  hazard  rate  z(ti)  ~  4>[N  —  {>  —  I)]t,  [73:105,112],  This  hazard  rate  is  similar  to  that 
of  e(|uation  2.3.  A  modified  version  was  subsequently  proposed  with  a  hazard  rate  of  r(t,)  = 
(?[.V  -  (»  -  !)][-«!?  -I-  6f,  +  c]  [73:112]. 

Asmimplions.  The  error  rate  is  not  constant,  and  errors  are  corrected  as 
soon  as  they  are  detected- “As  errors  occur,  the  routines  are  stopped,  the  error  is  identified,  cor¬ 
rected,  and  the  error  modality  is  reduced”  [73:111],  Musa  et  al.  classify  the  Schick- Wolverton 
models  as  finite  failure  Weibull  and  modified  Weibull  class  models,  which  assumes  that  at  infinite 
lime  the  number  of  failures  «>.\perienced  is  finite  [64:281-283],  Musa  el  al.  stale  that  for  the  modi¬ 
fied  model,  “It  does  not  appear  to  have  practical  applicability,”  and  also  that  “it  is  more  complex 
than  the  other  models  "  with  “no  evidence  of  superior  properties  that  would  justify  the  complexity” 
[64:283], 


2.2..(  Failure  Count  Models.  In  contrast  to  the  times  between  failures  models,  the  failure 
count  model  assumptions  are  ba.sed  on  test  interval  and  not  failure  interval  times  [34:1418-1419]: 

•  The  number  of  failures  discovered  during  a  t«st  interval  is  independent  of  the  number  discov¬ 
ered  during  a  different  nonoverlapping  test  interval. 

•  Testing  is  similar  and  uniform  throughout  the  different  test  intervals. 

•  Each  test  interval  is  independent  of  the  others. 

•  The  software  system  failure  rate  decreases  as  testing  proceeds. 

A  discussion  with  respect  to  the  major  models  follows. 
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Goel-Okumoto  Nonhomogeneous  Poisson  Process  Model. 

Equation.  The  general  equation  for  the  Goel-Okumoto  Nonhomogeneous 
Poisson  Process  (NHPP)  model  is  [35] 

P{N{t)  =  y}=  j/  =  o,  1.2,...  (2-6) 

with  rn(t)  =  a(l  — e~*')  and  A(/)  =  mi{t)  =  abe~'’‘  where  the  cumulative  number  of  failures  at  time 
t  is  denoted  by  N{t),  »«(<)  represents  the  expected  number  of  failures  at  time  /,  the  failure  rate  is 
A(<),  a  is  the  eventual  expected  number  of  failures,  and  6  is  the  fault  detection  rale  per  fault  (34). 

Assumptions.  The  number  of  failures  is  0  at  time  t  =  0,  and  the  number 
of  failures  occurring  during  nonoverlapping  time  intervals  are  mutually  exclusive  [35:206].  Also, 
the  number  of  remaining  faults  to  be  discovered  is  considered  a  variable  of  test  and  environmental 
factors  instead  of  a  fixed  constant  [34:1415].  This  is  considered  a  finite  failure  exponential  class 
model  [64], 


Musa  Execution  Time  Model. 

Equation.  Musa’s  Execution  Time  model  has  a  hazard  rate  of  [60:314] 

c(r)  =  JKNo  -  fl\v  (2.7) 

where  r  is  the  execution  time,  /  is  linear  execution  frequency  (instruction  execution  rate  per  number 
of  program  instructions),  I\  is  the  fault  expo.sure  ratio  (as  the  machine  state  may  vary,  this  accounts 
for  the  probability  of  a  fault  being  exposed  when  the  related  instruction  is  being  executed),  Ae  is 
the  number  of  inherent  errors  in  the  program,  and  n  is  the  number  of  faults  corrected  during  time 
T  [60].  This  concept  has  also  been  applied  to  the  determination  of  failures  experienced  (//)  for  a 
given  execution  time  (r)  [64:37] 


(j(t)  -  J/fl 


(2.8) 


as  well  as  the  measurement  of  current  failure  intensity  (A)  based  on  either  execution  lime  (r)  as 
shown  in  Equation  1.3  [64:39] 

A(r)  =  Aoexp  (2.9) 
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or  actual  failures  experienced  (/i)  [64:33] 


X(n)  =  Ao  (2.10) 

Here,  t^o  is  the  total  expected  number  of  failures,  and  Aq  is  the  initial  failure  intensity  (failures  per 
unit  time)  [64:528-530]. 

Assumptions.  The  basic  execution  time  model  has  been  around  for  quite 
some  time,  and  is  actually  considered  a  Poisson  process  model  [34,  60,  64],  This  model  assumes 
that:  program  faults  are  independent;  the  “potential  test  space  ‘covers’  its  use  space,”  not  in  a 
completeness  sense  but  rather  the  test  sets  should  be  representative  of  operational  program  u.se;  test 
inputs  are  randomly  selected;  all  failures  are  observed;  and  discovered  faults  are  corrected  before 
continuing  with  testing  or  are  not  counted  again  if  rediscovered  [60:313].  This  model  is  considered 
a  finite  failure  exponential  class  model  [64]. 

Musa-Okumoto  Logarithmic  Poisson  Execution  Time  Model. 


Equation.  The  Musa-Okumoto  Logarithmic  Poisson  Execution  Time  model 
is  expressed  by  [62:231] 

t<(’')  =  ^  •  in(Ao^T -1- 1)  (2.11) 

Here.  Aq  is  the  initial  failure  intensity,  and  0  is  the  failure  intensity  decay  parameter,  identifying 
how  fast  the  failure  intensity  is  changing  [62],  Again,  fi  is  the  number  of  failures  expected  for  a 
given  execution  time  r  [64:530-531],  As  with  the  Musa  Execution  Time  model,  measurement  of 
current  failure  intensity  (A)  can  be  made  from  either  execution  time  (r)  [64:39] 


A(-) 


Aq 

A(iflr  -|-  1 


(2.12) 


or  actual  failures  experienced  (//)  [64:34] 

A(^)  =  Aoexp(-^/r) 


(2.13) 


Assumptions.  This  model  uses  the  same  assumption  as  the  Goel-Okumoto 
.NHPP  model  in  Equation  2.6  with  respect  to  time  r  =  0:  however,  the  Musa-Okumoto  Logarithmic 
Poisson  Execution  Time  model  also  assumes  an  exponentially  decreasing  failure  intensity  based  on 
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the  number  of  failures  experienced  [62:230].  The  model  also  uses  t  to  determine  the  function  of 
the  mean  value  of  experienced  failures  with  respect  to  time  [62:231].  This  is  considered  a  geometric 
family  model  [64]. 

Shooman  Exponential  Model. 

Equation.  The  Shooman  Exponential  model  is  given  as  [76] 

(2.14) 

It 

where  p(r)  is  the  number  of  errors  per  total  number  of  instructions  detected  per  montli,  r  is  the 
number  of  months  after  start  of  system  test,  ki  is  the  proportionality  constant,  Et  is  the  total 
number  of  errors  (a  constant),  and  It  is  the  number  of  program  instructions  [76]. 

.Assumptions.  The  Shooman  model  uses  the  history  of  other  similar  soft¬ 
ware  programs  as  a  basis  for  determining  the  model  constants  [76:486].  This  model  assumes  “the 
total  number  of  errors  in  the  program  is  fixed”  and  the  number  of  errors  remaining  is  the  differ¬ 
ence  between  total  errors  and  errors  encountered  [76:487].  It  also  a.ssumes  “all  detected  errors  are 
corrected  errors,”  while  also  taking  into  account  that  “in  any  sizable  program  it  is  impossible  to 
remove  all  errors"  [76:488].  Another  assumption  is  both  the  number  of  debugged  errors  and  number 
of  errors  present  shouki  decrea.se  as  testing  proceeds  [76:492].  This,  taken  with  the  initial  assump¬ 
tion  that  errors  detected  are  proportional  to  the  number  present,  results  in  an  exponential  error 
detection  rate  [76:492].  Musa  et  al.  categorize  the  Shooman  model  as  a  finite  failure  exponential 
class  model  [64]. 


Yumada-Ohba-Osaki  Power  Model. 

Equation.  The  Vamada-Ohba-Osaki  Power  model  (also  referred  to  as  the 
5-Shaped  model)  is  a  NUPP  model  with  the  following  mean  value  function  for  time  t  [96:476] 

•'/(O  =  „[l  -  (1  +  6/)f-'"]  «,/;>0  (2.1.^) 

where  a  is  the  total  number  of  errors  and  b  is  the  error  detection  rate  [96:47-5].  Additionally,  the 
failure  intensity  is  given  by  [96:476] 
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A(/)  = 

A(0)  =  0 
A(oc)  =  0 

with  llie  remaining  expected  number  of  errors  determined  by  [96:476] 

n(t)  =  a(l  +6/)e““ 


Assvmpiions.  Tliis  model  assumes  a  steady-state  for  tlie  error  detection 
rate  b  [96:475].  Otlier  assumptions  include  random  occurrence  of  failures,  the  time  to  failure  —  I) 
impacts  the  time  to  failure  b  from  failure  {k  —  1),  prompt  correction  of  error(s)  each  time  a  failure 
occurs,  and  perfect  debugging  [96:475-476].  This  model  is  considered  a  gamma  class  Poi.s.son  finite 
failure  model  [64]. 

2.3  Svmtuary 

This  chapter  started  with  the  identification  of  software  quality  as  a  desirable  result  of  software 
engineering.  Softsvare  reliability  was  then  described  as  one  of  several  software  (|uality  factors 
that  affects  software  life-cycle  cost.  Next,  we  proceeded  to  identify  software  reliability  model 
cla.ssifications  within  the  scope  of  software  reliability  measurement.  As  many  papers  on  software 
reliability  exist,  it  was  necessary  to  define  the  overall  framework  for  software  reliability  model 
evaluation  l>efore  choosing  specific  models.  We  compared  and  contrasted  different  categories  of 
software  reliability  models.  The  baseline  framework  was  derived  from  a  synthesis  of  categories, 
primarily  following  the  RADC  and  Goel  categories.  Within  each  of  the  framework  major  categories, 
specific  software  reliability  models  were  then  identified  forevaluation.  The  evaluation  of  the.se  major 
models  is  descrilied  in  ('hapter  5. 
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III.  Software  Reliability  Model  Selection 


Tills  chapter  identifies  the  selection  of  the  candidate  software  reliability  models.  It  begins 
with  identification  and  discussion  of  the  software  reliability  model  selection  criteria.  The  criteria  are 
then  applied  to  select  candidate  software  reliability  models  for  evaluation  against  software  maturity 
data. 

3.1  Model  Selection  Criteria  and  Dtscnsston 

The  goal  of  this  thesis  is  to  identify  one  model  and  methodology  that  is  appropriate  for 
use  in  the  lOTicE  phase.  Toward  this  end,  the  criteria  defined  in  [39]  and  [d4].  as  well  as  other 
implementation  specific  criteria  defined  in  [16]  will  be  used;  however,  an  initial  screening  ba-sed 
on  model  requirements  eliminates  the  two  categories  of  fault  seeding  and  input  domain.  Mill's 
Hypergeometric  model  requires  fault  seeding  of  intentional  changes  to  the  software.  Such  seeding 
is  very  difficult  and  could  be  disastrous  for  .something  complex  like  avionics  flight  software.  As 
such  intentional  errors  are  not  something  to  be  introduced  after  the  start  of  10TA.:E.  this  model 
will  not  be  considered.  Similarly,  the  Ramamoorlhy-Bastani  model  will  not  be  considered.  The 
lOTfcE  input  domain  for  testing  is  based  on  operational  usage,  which  is  supported  by  the  model'.x 
lack  of  random  sampling  assumption;  however,  the  cost  of  determining  equivalence  classes  for  an 
ititegrated  weapon  system  (such  as  a  missile  or  aircraft)  would  be  prohibitive.  This  leaves  only 
the  failure  count  and  times-between-failure  models.  These  models  are  discussed  below  with  respect 
to  the  criteria  of  predictive  validity,  capability,  quality  of  assum|)tion.s.  applicability  to  the  finite¬ 
time  environment,  simplicity  of  design,  diversity  and  applicability  of  output,  and  capability  to  use 
existing  data. 

3.1.1  Predtcliri  Validity.  This  criterion  concerns  the  accuracy  of  a  model's  parameter  esti¬ 
mation.  and  not  t  he  predict  ion  of  the  reliability  itself  [64].  .\s  such,  predictive  validity  is 


the  cafiiihilily  of  the’  moiled  to  i>re<lict  future  failure  behavior  ctiirin;'  either  the  test  or 
the  operational  phases  from  present  and  past  failure  behavior  in  the  respe‘ctive  phase 
[39]. 

With  respect  to  a  “weighted  parameter  estimation"  of  number  of  errors,  both  the  Littlewood- 
Verrall  model  of  the  inverse  polynomial  family  and  the  Musa-Okumoto  Logarithmic  Poisson  Exe¬ 
cution  Time  model  were  more  accurate  in  the  first  60%  of  testing  than  the  Musa  Execution  Time 
model,  the  ^’amada-Ohba-Osaki  Power  model,  or  the  Crow  model  (described  as  a  power  family 
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Poisson  model  in  [64])  [89;9].  After  tliis  initial  phase,  all  of  these  models  performed  satisfactorily 
[89:9].  Of  the  models  analyzed  by  Musa  et  al.  in  [64],  the  geometric  and  inverse  polynomial  families 
had  the  best  initial  predictive  validity.  This  assessment  was  made  against  the  different  classes  and 
families  (the  type,  binomial  or  Poisson,  made  no  difference),  and  was  based  on  both  maximum 
likelihood  estimation  (MLE)  and  least  squares  estimation  (LSE)  [64:390].  Musa  et  al.  determined 
the  Miisa-Okumoto  Logarithmic  Poisson  Execution  Time  model  as  being  superior;  however,  the 
Musa  Execution  Time  model  becomes  just  as  viable  after  the  initial  60%  of  testing  [64:398].  The 
applicability  of  an  exponential  class  model  is  important,  as  software  maturity  data,  which  this 
thesis  suggests  could  be  the  basis  for  parameter  estimation,  has  historically  been  exponential  [94]. 

In  another  study  involving  16  data  sets  on  various  hardware  platfo-nis,  Angus  et  al.  found 
it  difficult  to  estimate  parameters  for  the  Jelinski-Moranda  and  Schick-Wolvcrton  models  [4:195]. 
While  the  Jelinski-Moranda  and  Schick-Wolverton  models  are  considered  finite  failure  models, 
both  geometric  and  inverse  polynomial  families  are  in  the  infinite  failure  category  [64:251].  Thus,  it 
appears  that  it  is  easier  and  more  accurate  to  estimate  parameters  for  models  of  the  infinite  failure 
category,  as  opposed  to  the  finite  failure  category.  For  lOTAtE,  such  parameter  estimation  could 
be  based  heavily  on  data  previously  collected  prior  to  the  start  of  lOT&E  (either  on  the  system 
undergoing  test  or  from  another  similar  systent  that  has  completed  test).  Initial  parameters  could 
then  be  predicted  using  a  geometric  or  inverse  polynomial  model  that  is  Poisson  in  type. 

3.J.2  Capabthiy.  Another  criterion,  capahtlily.  is  defined  by  lannino  et  al.  as 

. . .  the  ability  of  the  inode!  to  estimate  with  satisfactory  accuracy  quantities  needed  by 
software  managers,  engineers,  and  users  in  planning  and  managing  software  development 
projects  or  running  operational  software  systems  [39]. 

Such  accuracy  of  estimate  could  then  be  measured  in  the  following  quantities  [64]: 

•  Present  reliability.  MTTI’,  and  failure  intensity. 

•  Expected  date  to  reach  specifietl  reliability,  .MTTF.  or  failure  intensity  objective. 

•  Human  and  computer  resource  and  cost  requirements  needed  to  reach  the  failure  intensity- 
objective. 

This  criterion  is  important  for  lOl'icE,  as  the  test  director  needs  to  know  both  the  current 
quality  of  the  software  and  what  it  will  cost  (in  time  and  money)  to  reach  an  acceptable  level  of 
quality.  Musa  et  al.  conducted  an  evaluation  and  comparison  of  18  major  software  reliability  models 
[64].  Of  the  18  models  examined,  those  of  the  exponential  cla.ss  and  geometric  family  appear  to 
have  the  best  capability  to  be  used  to  make  quality  assessments  of  the  software  under  test  [64]. 


3.1.3  Quality  of  Assumptions,  lanninoetai.  recommend  that  assumptions  should  be  tested; 
however,  if  this  is  not  possible,  the  assumption's  “plausibility”  should  be  considered  based  on 
logical  consistency  and  the  user’s  software  engineering  experience  [39],  For  complex  systems,  it 
is  difficult  to  test  the  validity  of  software  reliability  model  assumptions.  An  example  of  this  was 
the  Hughes  Joint  Surveillance  System  (JSS)  air  defense  system  for  North  America,  where  it  was 
not  possible  to  confirm  the  validity  or  lack  thereof  of  all  software  reliability  model  assumptions 
used  in  evaluating  the  software  [3:268,‘270].  As  lOTAcE  is  performed  on  weapon  systems  of  similar 
complexity  to  the  JSS,  there  will  be  no  attempt  to  prove  or  disprove  all  the  assumptions  for  the 
models  under  consideration.  Instead,  a  comparison  of  only  the  assumptions  deemed  necessary 
for  lOTicE  assessment  will  be  performed  against  the  models'  assumptions.  A  model  fails  this 
comparison  if  only  one  major  lOTA'E  assumption  is  not  supported  by  the  model’s  assumptions. 

Both  Musa  et  al.  [64]  and  Goel  [31]  identify  many  critical  assumptions  that  are  necessary  for 
model  implementation.  For  application  to  the  lOTAcE  environment,  the  major  assumptions  were 
derived  from  both  HQ  AFOTEC  requirements  and  the  author’s  experience  in  lOT&E  of  weapon 
system  software  and  include  [47]: 

1.  Operational  testing  is  representative  of  the  operational  environment. 

2.  There  is  imperfect  debugging  for  fault  removal. 

3.  Errors  might  not  be  corrected  after  the  test  interval  (i.e.-just  after  a  test  flight). 

4.  Execution  time  is  used  for  tlie  failure  rates. 

-Assumption  1  allows  both  times  l>etween  failures  and  failure  count  models  to  a.ssess  the  soft¬ 
ware  with  respect  to  operational  reliability  [34:1418],  The  assumption  is  based  on  the  operational 
profiles  used  to  assess  the  overall  performance  of  .system  testing  [2:1].  System  testing  is  the  usual 
level  of  test  for  a  Test  and  Evaluation  effort;  however,  there  is  usually  insufficient  test  lime  to 
thoroughly  test  all  the  .software  due  to  the  tremendous  combinatorics  that  occur  from  integrating 
even  the  simplest  subsystems  together  [48:110.114].  As  a  cons<'(|uence,  using  operational  profiles 
for  testing  differs  in  t  he  degree  of  randomness  (and  thus  thoroughness)  that  is  possible  with  module 
or  unit  level  testing.  .Since  the  test  ca.ses  are  then  not  likely  to  be  independent,  the  test  process  will 
not  follow  a  true  random  nature  [34:1417].  This  eliminates  limes  between  failure  models,  which 
assume  times  between  failures  occur  independently  [34:1417,1419].  In  addition,  this  assumption 
makes  an  important  contribution  to  determination  of  end  of  operational  testing  and  start  of  op¬ 
erations.  Since  lOTi'cE  testing  is  targeted  for  an  operational  environment,  a  final  lOTicE  value 
of  a  failure  intensity  would  then  be  the  constant  failure  intensity  expected  to  occur  throughout 
operations  until  the  next  major  software  release. 


Assumption  2  also  eliminates  the  Shooman  model,  and  most,  if  not  all,  of  the  times  between 
failures  category  models  [27:4-7], [3-4:1418-1419].  Ohba  and  Chou  have  assessed  the  validity  of  the 
perfect  debugging  assumption  found  for  the  times-between-failures  models,  noting  that  “software 
reliability  growth  models  sometimes  give  reasonable  figures  (fairly  accurate  estimations)  in  con¬ 
ditions  where  the  perfect  debugging  assumption  is  not  valid”  [67:41].  They  have  also  proposed 
modifications  to  the  Jelinski-Moranda  and  Goel-Okumoto  models  to  accommodate  imperfect  de¬ 
bugging;  however,  they  cite  that  further  study  using  actual  project  data  is  necessary  to  verify  the 
modified  models'  applicability  [67:45],  Goel  and  Okumoto  have  also  proposed  a  modified  model,  the 
Goel-Okumoto  Imperfect  Debugging  model,  which  is  an  e.vtension  of  the  Jelinski-Moranda  model 
based  on  a  Markov  process  [34:1414];  however,  this  model  is  eliminated  from  consideration  by  .As¬ 
sumptions  1  and  3.  Ohba  and  Chou  also  note  the  necessity  of  verifying  the  impact  of  an  imperfect 
debugging  assumption  on  5-shaped  software  reliability  models  (such  as  the  5’amada-Ohba-Osaki 
Power  model  discussed  in  Chapter  2)  before  concluding  that  the  imperfect  debugging  assumption 
does  not  affect  software  reliability  data  analysis  [67:46],  Until  such  proof  e.xists,  the  Yamada-Ohba- 
Osaki  Power  model  will  still  be  counted  under  the  perfect  debugging  assumption  and  thus  e.xcluded 
from  further  consideration  [96:476],  In  contrast,  failure  count  models,  such  as  the  Musa  E.xecution 
Time  model,  can  incorporate  imperfect  debugging  through  a  fault  reduction  factor  of  the  ratio  of 
net  number  of  faults  corrected  per  total  number  of  faults  corrected  [64:120],  .Musa  et  al.  suggest 
such  a  ratio  could  be  independent  of  specific  project  characteristics,  and  sufficient  values  have  been 
determined  to  provide  for  boundary  conditions  and  an  average  [64:120-121], 

■Assumption  3  further  eliminates  the  times  between  failures  models  and  the  A'amada-Ohha- 
Osaki  Power  model,  as  these  models  require  faults  to  be  removed  as  soon  as  they  are  detected 
[34:1419],  [96:476],  The  last  one.  Assumption  4,  is  important,  as  the  concept  of  lOTAiE  revolves 
around  the  time  (flight,  CPU,  etc.)  available  for  testing  within  given  monetary  constraints  [48:114], 
This  assumption  further  eliminates  fault  seeding  and  input  domain  models  (as  neither  define  pa¬ 
rameters  in  terms  of  time),  and  also  restricts  times  between  failures  and  failure  count  models  to 
their  e.xecutioti  insleatl  of  calendar  time  components. 

Applicdhilify  to  tli(  Fnnie-Ttme  Enrtronmeul.  Applicability  addresses  five  general 
categories  that  the  software  reliability  model  should  be  able  to  deal  with  [39]: 

•  Phased  ititegration  of  a  program  during  test  (result  is  that  initial  failure  data  is  based  on  only 
a  portion  of  the  program). 

•  Design  changes  to  the  program. 

•  Failure  severity  classification  using  different  categories. 


•  Ability  to  liandlc  incomplete  failure  data  or  data  with  measurement  uncertainties. 

•  Operation  of  the  same  program  on  computers  of  different  performance. 

Any  model  that  meets  these  should  then  have  the  capability  to  be  a  single  useful  model,  as 
well  as  something  that  will  be  applicable  across  different  lOT&E  efforts/systems.  Musa  et  al.  [04] 
identifies  the  characterist  ics  of  several  models  that  allow  for  dealing  with  these  categories.  Of  these 
models,  those  of  both  the  e.xponential  class  and  geometric  family  apply  well,  as  initial  parameters 
can  be  derived  from  data  that  exists  prior  to  program  testing  (such  as  software  size,  machine 
execution  rate,  etc.)  [til].  The.se  parameters  could  then  be  further  refined  through  data  collected 
on  any  evaluation,  such  as  software  maturity,  done  prior  to  the  start  of  lOl'irE. 

i.].5  //j/  of  D( sign.  Simplicity  should  be  present  in  t  hree  areas  [.'{9]: 

•  It  must  be  simple  and  inexpensive  to  collect  the  required  data. 

•  The  model  itself  should  be  simple  in  concept. 

•  The  model  must  be  implementable  as  both  a  useful  management  and  engineering  tool. 

The  Musa  Execution  Time  model  was  found  easy  to  use;  however,  would  generally  “under¬ 
estimate  the  number  of  errors”  [89:9].  In  addition  to  this  model,  the  Musa-Okumoto  Logarithmic 
Poisson  Execution  Time  model  was  also  identified  as  one  of  the  easiest  to  use  models  [1)4:398].  In 
contrast,  the  Cioel-Okiimoto  .NllF'P  and  Jelinski-.Moranda  models  were  found  to  havi' such  “numer¬ 
ical  difficulties”  that 


The  issues  concerning  starting  points  for  the  iterative  procedures,  uniqueness  of  the 
parameter  estimates,  and  even  alternative  estimation  techniques  must  be  studied  and 
such  jiroblems  solved  before  these  motlels  can  be  used  by  acquisition  managers  [3:273]. 

The  Lit  tIewood-Verrall  model  is  very  complex,  very  difficult  to  understand,  and  very  difficult 
to  ap|)ly  on  a  computer  [(>4:32].  Markov  models,  in  general,  were  also  found  to  have  a  “great  deal 
of  added  complexity"  with  ■much  research  still  needed  in  this  area”  [27:4-116].  In  contrast,  the 
Poisson  type  models  (of  the  exponential  class)  and  the  .Mu.sa-Okuinoto  Logarithmic  Pois.son  fime 
model  (of  the  geometric  family)  are  the  two  simplest  models  to  implement  [34,  64,  89]. 

■3.1.6  Dtversily  atid  Appitcabiltly  of  Ovipul.  The  ability  to  express  data  and  results  in  differ¬ 
ent  formats  is  desirable  considering  the  diversity  of  soft  ware  systems  that  undergo  lOT.VrE.  .\llowing 


the  data  to  be  presented  in  different  formats  will  allow  the  software  engineers/analysts  to  better 
convey  the  meaning  of  reliability  measurements. 

While  all  models  possess  the  capability  to  provide  meaningful  data  to  the  decision  makers, 
the  Poisson  type  and  basic  execution  time  models  have  the  potential  to  encompass  more  than 
just  the  raw  data.  Of  all  the  models  evaluated,  only  the  Musa  Execution  Time  model  and  the 
■Musa-Okamoto  Logarithmic  Poisson  Time  model  have  derived  equations  to  compute  current  fail¬ 
ure  intensity  as  a  function  of  either  failures  experienced  or  elapsed  test  time.  No  other  models 
have  straightforward  equations  to  determine  both  the  number  of  failures  or  amount  of  time  that 
is  expected  to  occur  before  reaching  a  desired  failure  intensity.  Additionally,  of  the  models  evalu¬ 
ated,  only  these  two  had  equations  to  relate  system  characteristics  to  the  determination  of  initial 
parameters.  Such  equations  allow  for  evaluation  of  a  system  where  previous  or  similar  failure  data 
do  not  exist.  These,  and  the  other  equations,  also  enable  presentation  of  data  ranging  in  form  from 
engineering  units  vs.  specific  system  parameters  to  overall  trends  of  failures  vs.  system  time. 

:i.J.  7  Capability  to  Use  Existing  Initial  Data.  The  criteria  of  simplicity  of  design  addresses 
the  ease  and  cost  of  collecting  data  for  the  reliability  model.  In  contrast,  the  capability  to  use 
existing  initial  data  evaluates  a  model’s  flexibility  to  be  mapped  to  an  existing  database.  HQ 
.AFOTEC  is  developing  a  database  of  software  failure  data  to  analyze  and  determine  the  software 
mat  urity  for  different  weapon  systems.  A  software  reliability  model  should  then  be  able  to  use  this 
initial  data  as  a  ba.seline  for  estimating  parameters.  Such  estimation  is  important,  and  using  initial 
ilata  can  reduce  errors  from  the  use  of  data  from  other  “similar”  systems. 

Some  Poi.sson  process  models  use  cumulative  failures  per  test  period  [34];  however,  the  use  of 
time  n/ failure  occurrence  and  not  time  betiveen  failure  occurrence  allows  for  modeling  the  failure 
occurrence  as  a  random  arrival  event  for  those  data  points  collected  without  time  information.  This 
process  has  been  demonstrated  in  [(>4],  and  can  be  useful  for  using  existing  maturity  data  where 
failures  i>er  test  lime  are  the  only  available  data. 


Li  CboiK  of  a  Rf  liability  Model 

Ba.sed  on  the  criteria  and  discussions  above,  the  following  models  can  he  dismissed  as  possible 
candidates  for  the  following  reasons; 

•  Mill's  Hypergeometric  Model.  This  and  any  other  fault  seeding  models  are  not  viable  for 
lOT&E  as  the  introduction  of  faults  this  late  in  software  testing  would  adversely  impact 
.system  delivery.  Seeding  such  faults  in  a  manner  to  be  representative  of  the  innate  faults  is 
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very  difficult,  and  is  not  practical  for  lOTicE  of  programs  with  extensive  amounts  of  software. 
Also,  the  model  does  not  support  the  use  of  execution  times  for  failure  rates. 

•  Ramamoorthy-Bastani  Model.  Input  domain  models  are  not  workable  due  to  the  high  cost  of 
determining  equivalence  classes.  Also,  the  model  does  not  support  the  use  of  execution  times 
for  failure  rates. 

•  Jelinski-Moranda  Model.  Parameter  estimation  was  found  to  be  difficult.  The  model  does  not 
support  lOT&E  assumptions  of  imperfect  debugging  for  fault  removal  or  errors  not  corrected 
immediately  after  a  test  interval.  It  is  one  of  the  more  difficult  models  (numerically)  to  use. 

•  Littlewood-Verrall  Model.  The  model  does  not  support  IOTA;E  assumptions  of  imperfect 
debugging  for  fault  removal  or  errors  not  corrected  immediately  after  a  test  interval.  This 
model  is  also  very  complex  atid  difficult  to  understand  and  apply  on  a  computer. 

•  Sell ick-Wolverton  Model.  Parameter  estimation  was  found  to  be  difficult.  The  model  does  not 
support  lOT&E  assumptions  of  imperfect  debugging  for  fault  removal  or  errors  not  corrected 
immediately  after  a  test  interval. 

•  Goel-Okumoto  NHPP  Model.  With  respect  to  obtaining  parameter  estimates,  it  is  one  of  the 
more  difficult  models  to  use.  Also,  this  model  does  not  support  the  lOTicE  assumption  of 
imperfect  debugging. 

•  Shooman  Exponential  Model.  This  model  does  not  support  the  lOTikrE  assumption  of  im¬ 
perfect  debugging.  The  model  also  relies  on  calendar  time  and  not  execution  time. 

•  Vamada-Ohba-Osaki  Power  Model.  Accuracy  of  parameter  estimation  not  acceptable  until 
approximately  60%  into  testing.  The  model  does  not  support  lOTi'cE  assumptions  of  imper¬ 
fect  debugging  for  fault  removal  or  errors  not  corrected  immediately  after  a  test  interval. 

Therefore,  only  two  models  from  the  failure  count  category  were  .selected  as  candidate  models 

evaluation: 

•  .Musa-Okumoto  Logarithmic  Poisson  Execution  Time  .Model.  This  model  was  found  to  have 
the  best  initial  predictive  validity  for  parameter  e.stimation.  as  well  as  the  best  capability 
to  be  used  to  make  software  assessments.  The  model  supports  all  10Ti:E  assumptions  and 
easily  accommodates  diverse  output.  It  can  also  use  existing  program  data  to  determine 
initial  model  parameters. 

•  Musa  Execution  Time  Model.  This  model  was  found  to  be  one  of  the  models  having  the  best 
capability  to  be  used  to  make  .software  assessments.  The  model  also  supports  all  lOTAiE 


assumptions  and  easily  accommodates  diverse  output.  This  model  can  also  use  existing 
program  data  to  determine  initial  model  parameters. 

Although  the  Musa  Execution  Time  model  does  not  support  adequate  parameter  estimation 
until  60%  of  testing  is  complete,  this  assessment  is  based  on  accumulated  failure  data  and  not 
existing  program  data.  .‘\s  the  model  is  one  of  the  simpler  ones  to  implement,  it  is  hoped  that  the 
simplicity  and  capability  to  use  existing  program  statistics  will  enable  closer  parameter  determina¬ 
tion  than  is  possible  with  using  failure  data  alone.  The  Musa  Execution  Time  model  also  contains 
the  salient  points  from  other  models,  such  as  the  Goel-Okumoto  NHPP  model,  Jelinski-Moranda 
model,  and  Shooman  Exponential  model  [64:32].  One  comparison  even  stated  the  Musa  Execution 
Time  model  and  Jelinski-Moranda  model  were  equivalent,  with  the  Musa  Execution  Time  moilel 
considered  to  be  “better  developed"  of  the  two  [6:15].  Similarly,  the  Mnsa-Okumoto  Logarithmic 
Poisson  Execution  Time  model  is  considered  a  combination  of  the  Musa  Execution  Time  model's 
execution  time  characteristic  and  the  “analytical  ease"  of  the  Goel-Okumoto  NHPP  model  [58:83]. 
Other  failure  count  models  are  similar  to  the  candidate  models,  either  being  more  generalized  or 
more  refined  for  specific  applications  [34,  64,  88,  96]. 

The  final  two  selection  criteria  have  additional  impact  on  the  implementation  of  these  can¬ 
didate  models.  Several  tools  exist  which  can  assess  software  reliability  with  respect  to  different 
models  [28,  S3];  however,  the  thrust  of  these  tools  (and  hence  the  model  implementation)  is  to  pre¬ 
dict  software  reliability  [83:1],  To  fully  examine  the  current  assessment  capability  of  the  candidate 
models,  a  fresh  implementation  must  be  considered.  This  implementation  is  discussed  in  the  next 
chapter. 


.3..;?  Summary 

This  chapter  took  the  models  described  in  Chapter  2  and  compared  them  against  specific 
motlel  selection  criteria,  with  the  goal  of  selecting  one  candidate  model  and  methodology  appro¬ 
priate  for  'use  in  the  lOTA'E  phase  of  software  test  and  evaluation.  The  model  selection  criteria 
were  defined,  and  models  wen*  either  vindicated  or  eliminated  during  the  discussion  of  each  cri¬ 
terion.  The  results  were  two,  instead  of  a  single  one.  software  reliability  models  that  should  be 
appropriate  for  software  lOTA’E;  the  Musa  Execution  Time  model;  and  the  Musa-Okumoto  Log¬ 
arithmic  Poisson  Execution  Time  model.  The  implementation  of  these  models  is  described  in  the 
next  chapter. 
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IV.  Software  Reliability  Model  Implementation 


This  chapter  contains  the  method  and  actual  implementation  of  the  candidate  models  identi¬ 
fied  in  the  previous  chapter.  Several  software  reliability  implementation  methodologies  have  been 
presented,  including  [34,  41,  64,  71].  The  salient  points  of  each  have  been  extracted  and  are  used 
as  a  basis  for  implementation  of  the  candidate  models: 

•  Plan  a  strategy  [41:33-35]. 

•  Determine  software  reliability  goals  [41:35]. 

•  Assess  existing  data  [34:1420]. 

•  Select  candidate  inodel(s)  [34:1420]. 

•  Derive  fitted  model  [34: 1420], [71:50]. 

•  As.sess  the  model  [34:1420], [71:50]. 

•  Define  and  implement  data  collection  procedures  [41:35], [64:215-220]. 

•  Assess  the  software  reliability  [34:1421], [41:36], [71:50]. 

A  discussion  of  each  follows. 

4.1  Plan  a  Stralfgy 

This  step  is  defined  as  “initiate  a  planning  process"  [41:33],  and  will  be  performed  at  two 
levels.  First,  software  reliability  needs  to  be  incorporated  into  the  lOTA’E  test  planning  strategy. 
After  that,  the  design  and  implementation  of  the  candidate  models  will  follow. 

4.].!  lOT&E  Test  Planning  Stralfgy.  With  respect  to  the  overall  OTi^E  test  planning 
strategy. 


Operational  Test  and  Evaluation  (OTA:E)  is  conducted  to  estimate  the  system's  opera¬ 
tional  effectivcne.ss.  operational  suitability  (including  Teliability.  availability,  maintain¬ 
ability.  logistics  supportability,  and  training  requirements)  and  identify  needed  modifi¬ 
cation  [21:3-4]. 

As  the  premise  of  this  thesis  is  that  software  maturity  data  can  be  used  as  a  basis  for  initial 
paranteters  of  the  software  reliability  mea.surement.  the  candidate  models  must  be  implemented, 
where  possible,  o/ter software  mattirity  data  has  been  collected. 
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Figure  4.1.  Software  TA;E  with  Software  Reliability  Assessment 


Figure  4.1  indicates  one  possible  method  for  integrating  software  reliability  measurement  into 
the  lOTicE  test  effort.  This  figure  identifies  a  possible  relationship  of  software  Ti:E  during  both 
the  developmental  TIE  (DTicE)  and  OT&E  phases.  This  method  integrates  software  reliability 
evaluation  w  ith  current  HQ  AFOTEC  operational  suitability  assessments  (software  maintainability, 
usability,  maturity,  and  support  resources),  and  makes  use  of  historical  data  for  the  same  weapon 
system  collected  by  software  evaluation  personnel  prior  to  the  start  of  lOTikrE.  Such  a  combined 
approach  shouki  provide  a  quantifiable  way  of  assessing  whether  or  not  the  soon-to-be  operational 
system  has  "good  code." 

4  J.2  Program  Design  Strategy.  The  development  plan  for  this  software  effort  involved  an 
analysis  of  the  problem,  specification  of  requirements,  and  development  of  a  design  ba-sed  on  the 
ifquiremenls.  .After  this,  coile  development  and  testing  followed.  While  the  waterfall  model  pro- 
vidi's  the  structure  for  this  type  of  effort,  an  iterative  waterfall  (or  "waterfountain" )  approach  was 
used  to  enable  further  refinement  of  the  specifications  prior  to  generation  of  data  sets  [84]. 

Structured  analysis  techniques  were  used  for  the  initial  analysis.  The  resultant  data  flow- 
diagrams  (DFDs)  were  used  for  an  object-orienleil  design  of  the  software.  As  part  of  the  high- 
level  design  of  the  system,  the  possibility  of  using  an  abstract  data  type  (.ADT)  to  implement  the 
software  was  considered.  Program  coding  was  done  in  the  Clipper  programming  language,  which  is 
a  dBASE  compiler  for  any  computer  capable  of  running  at  lea.st  PC/MS-DOS  version  2.0  [66:1-4]. 
The  Clipper  language  was  chosen  for  compatibility,  as  the  current  software  maturity  data  base  and 
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supporting  software  were  all  previously  developed  using  Clipper.  Testing  of  the  code  was  performed 
throughout  the  software  life-cycle  effort.  Specific  details  on  the  analysis  and  design  are  discussed 
in  Appendix  C. 


4-2  Delermine  Software  Reliability  Goals 


The  software  reliability  goals  of  this  thesis  are  not  to  predict  software  reliability  at  any  time 
in  the  future.  Instead,  the  goal  is  to  be  able  to  define  a  current  measure  of  the  software  such  that 
a  decision  maker  (the  Test  Director  for  lOTiiE  testing)  may  be  able  to  assess  how  much  longer 
it  will  take  or  how  many  more  failures  will  be  discovered  to  reach  a  failure  intensity  objective  of 
his/her  choosing.  Typical  values  for  operational  reliability  of  critical  software  systems  (such  as  air 
traffic  control  systems,  nuclear  power  plants,  and  space  systems)  have  ranged  from  10“'  failures 
per  CPU  hour  to  10“®  failures  per  CPU  hour  [04:93].  Another  suggested  value  is  a  reliability  of 
0.999999  for  a  mission  duration  of  5  hours  [71:50].  Therefore,  the  suggested  reliability  goals  will  be 
0.999999,  0.9999,  0.99,  0.95,  0.90,  0.85,  and  0.80,  all  of  which  are  within  the  range  [0,1]. 

In  order  to  determine  which  of  these  is  the  optimum  reliability  goal,  there  are  two  concepts 
that  must  be  considered:  failure  intensity  at  the  end  of  lOT&cE  is  the  same  as  that  for  beginning 
of  the  software’s  operational  life;  and  given  an  unchanging  failure  intensity  during  operations, 
different  reliability  values  for  operational  periods  can  be  used  to  assess  the  software  reliability  at 
end  of  10Ti:E.  While  this  might  seem  like  a  back-door  method,  it  does  have  some  merit  given 
that  engineers  can  not  determine  (with  any  degree  of  accuracy)  the  future  reliability  of  software 
in  major  weapon  systems.  Thus,  the  decision  maker  should  be  able  to  pick  a  desired  operational 
reliability  (with  respect  to  failure  intensity),  with  the  engineer  then  assessing  the  cost  to  reach  that 
goal.  This  follows  the  concept  that  an  acceptable  range  of  reliability  values  should  be  established, 
given  the  user's  requirements  and  needs  [41:35], 


In  specifying  the  user's  requirements,  we  will  start  with  the  basic  reliability  ftinction,  R{t), 


which  is  given  by  [38:524] 


/?(<)=  1-F(0  = 


where  t  is  the  lime  of  relialulity  assessment,  F{t)  is  the  cumulative  distribution  function  for  failures, 
and  f{j-)  is  the  probability  density  function  for  failures  [38:54,56,524].  .Assuming  only  random 
failures  are  used  (this  gives  an  exponential  time  to  the  failure  density),  the  reliability  function  is 
described  in  terms  of  a  Poisson  distribution  with  a  mean  occurrence  rate  A  by  [38:524,526] 


Musa  et  al.  applies  this  to  software  reliability,  resulting  in  a  similar  reliability  function  R(t)  given 
by  [64:50] 

R(T)-e-^''  (4.1) 

The  major  assumption  for  this  is  a  constant  failure  intensity  A  for  the  execution  time  period  r 
[64:50].  However,  this  works  to  the  advantage  of  the  decision  maker.  Taking  the  natural  logarithm 
of  Equation  4.1  gives 

ln(/?(r))  =  -Ar  (4.2) 


Equation  4.2  can  then  be  used  for  decision  support  alternatives.  For  example,  assuming  a 
weapon  system  is  projected  to  operate  for  (an  average)  of  500  hours  per  each  calendar  year,  t  he 
Test  Director  would  pick  the  reliability  goal  and  the  required  failure  intensity  from  the  range  of 
values  derived  for  various  A  values  specified  above  (see  Table  4.1).  The  reliability  would  be  defined 
by  the  Test  Director  as  a  success  criterion,  and  the  implemented  model  should  be  able  to  support 
analysis  based  on  current  operational  assessment  as  well  as  a  potentially  changing  success  criterion. 
In  this  case,  the  additional  test  time  needed  to  reach  the  desired  failure  intensity  (determined  from 
the  reliability  defined  by  the  Test  Director)  would  then  be  calculated. 

Table  4.1.  Range  of  Software  Reliability  Goals  for  r  =  500  Hours 


A 

R(bOO) 

2.00  X  10"^  Failures/Hr 

0.999999 

2.00  X  10“'  Failures/Hr 

0.9999 

2.01  X  10“®  Failures/Hr 

0.99 

1.03  X  10“'*  Failures/Hr 

0.95 

2.11  X  10“'*  Failures/Hr 

0.90 

3.25  X  10“'*  Failures/Hr 

0.85 

4.46  X  10“'*  Failures/Hr 

0.80 

This  is  supported  by  the  candidate  models,  as  predicted  and  measured  quantities  (number 
of  failures  remaining  and  mean  time  to  fail,  respectively)  at  the  start  of  operations  “are  constant 
and  equal  to  those  at  the  end  of  the  last  test  phase  (unless  errors  are  corrected,  in  which  ca.se  the 
operational  phase  should  be  considered  as  a  ‘test’  phase  or  phase  of  reliability  growth)"'  [60:;ii:!]. 
Thus,  the  desired  final  reliability  value  (A^-)  is  determined  from  Equation  4.2,  and  the  present 
failure  intensity  during  10TA:E  testing  (Xp)  is  determined  from  either  Equations  2.9  and  2.10  or 
Equations  2.12  .and  2.13.  The  amount  of  additional  test  time  (At)  necessary  to  reach  the  desired 
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software  reliability  level  is  then  determined  by  the  Musa  Execution  Time  model  from  [64:45] 


Ar  = 


(4.3) 


and  by  the  Musa-Okumoto  Logarithmic  Poisson  Execution  Time  model  from  [64:45] 


At  = 


(4.4) 


Therefore,  if  the  test  time  needed  to  reach  a  desired  failure  intensity  objective  was  deemed 
to  be  too  much  by  the  Test  Director,  he/she  would  then  have  to  choose  a  lower  reliability  goal, 
obtain  additional  test  time,  or  alter  some  other  aspect  of  the  software  development  process  to 
compensate.  Thus,  the  actual  software  reliability  goals  will  be  determined  by  the  decision  maker, 
and  are  subject  to  change  based  on  the  availability  of  test  resources  (primarily  time).  This  means 
the  implementation  must  support  some  form  of  decision  support  scenario. 


/Issess  Eiisiiiig  Data. 
Shaw  noted 


The  problem  in  applying  software  metrics  is  to  find  appropriate  measures  and  make 
sense  out  of  the  data,  not  simply  to  obtain  the  data  [75:257], 

The  goal  of  software  reliability  as,sessment  is  to  make  the  data  useful,  thus  something  must  be 
determined  from  the  data,  even  if  that  means  discovering  that  nothing  can  be  determined  from 
the  data.  From  the  HQ  AFOTEC  .software  maturity  data.  17  initial  data  sets  were  available  that 
included  aircraft,  communications,  missile,  radar,  and  space  systems.  For  thes*'  data  sets,  the 
number  and  type  of  record  fields  varied;  however,  there  was  a  common  set  of  fields  across  all  17 
data  sets.  These  fields  are  identified  in  Table  4.2. 

None  of  the  data  in  the  17  different  data  base  files  contained  information  about  test  durations 
or  specific  descriptions  of  the  system  under  test  (for  example,  number  of  source  lines  of  code  or 
processor  execution  rate).  Such  additional  information  was  necessary  to  run  the  models;  however, 
due  to  the  very  recent  incorporation  of  software  maturity  assessment  in  the  IOTA:E  planning 
strategy,  this  initial  data  was  “fragmented  and  incomplete”  [45],  Therefore,  a  data  assessment 
strategy  was  devised  where  candidate  data  sets  were  chosen  based  on  the  availability  of  any  test 


Table  4.2.  Common  Software  Maturity  Data  Fields 


Field  Name 

Description 

Date 

Date  of  Problem 

CPCl 

Software  Configuration  Item 

Sev.Code 

Severity  of  Problem 

Date-Fix 

Date  Problem  Fixed 

Title 

Description  of  Problem 

Prob-Num 

Software  Problem  Number 

duration  data.  Tliis  limited  the  data  sets  to  three  types  of  weapon  systems:  aircraft  (denoted 
by  .\),  space  systems  (denoted  by  S),  and  weapon  system  trainers  (denoted  by  \V).  These  data 
sets  were  then  plotted  with  failure  count  indicated  as  a  function  of  execution  time  [34:1420].  An 
as.sessment  was  then  made  as  to  the  applicability  of  the  candidate  models  based  on  the  initial  curve 
of  the  data.  The  results  of  this,  as  well  as  the  application  of  the  models  to  the  data,  are  discussed 
in  the  next  chapter. 

Stieciion  of  Candidate  Models. 

Assumptions  for  each  model,  evaluation  of  each  model  with  respect  to  specific  acceptance 
criteria,  and  selection  of  candidate  models  were  discussed  in  the  previous  chapter. 

T-t  Derive  the  Fitted  Model 

This  procedure  involves  botii  estimating  the  parameters  for  the  model,  and  then  substituting 
the'se  parameters  into  the  model  to  fit  the  model  for  the  data  [34:1420].  .-\n  additional  version 
of  each  fitted  moilel  was  derived  for  those  models  that  had  prior  DTA:F  test  data.  A  discussion 
of  initial  parameter  estimates  apjtears  in  the  first  section,  followed  by  a  discussion  of  the  derived 
parameter  t'stimates. 

.',..5.1  .Model  Paravitte  r  F.stiwnhon.  Musa  et  al.  define  equations  for  failure  ititensity  and 
mean  value  functions  for  hot  It  tlie  Execution  Time  model  and  bogarithmic  Poisson  Execution 
Time  model  (see  Table  4.3)  [64:307].  From  these,  the  parameters  if,  =  i/q  (the  total  failures  at  time 
t  =  iX  for  the  Execution  Time  model)  and  /?,7'  =  0  (the  failure  intensity  decay  parameter  for  the 
Logarithmic  Poisson  Execution  Time  model)  need  to  be  determined  [64:351],  Parameter  00,  as  well 
as  other  estimated  values,  are  a  function  of  which  is  defined  as  0\  =  Ao/z/o  for  the  Execution 
Time  model,  and  0\  =  for  the  I.ogarithmic  Poisson  Execution  Time  model  [61:351,529].  The 


Table  “1.3.  S|>ecific  Model  A  and  p  Functions 


Model 

Execution  Time 

/?o[l-e-^>‘] 

Logarithmic  Poisson 
Execution  Time 

i!?0  hi(  1  -|-  ^i<) 

Po/3, 

parameter  /i)  itself  is  estimated  for  tlie  Execution  Time  model  by  [64:325] 


lllf  lV(lf 


-£/.=0 


(T5) 


and  is  estimated  for  the  Logarithmic  Poisson  Execution  Time  model  by  [64:326] 


1  ^  1 


ITIfte 


y 

\  (1  +/?,te)ln(l  +A<e) 


=  0 


(4,6) 


4.5. 1. 1  Niwton-Rapbsov  Mtihod.  One  way  of  estimating  parameters  is  with  the  Newion- 
Raphson  method,  which  has  the  general  form  [14:48] 


}>n  —  /'ti-l  ^ 

This  is  calculated  based  on  a  simple  algorithm,  such  as  the  one  presenteil  by  Burden  and  Faints 
[14:49]: 


To  find  a  solution  to  f{r)  =  0  given  an  initial  approximation  po: 

INPUT  initial  aiiproximation  po;  tolerance  TOL;  maximum  number  of  iteration  .V(,. 
OUTPUT  approximati’  solut  ion  p  or  message  of  failure. 

Step  I.  Set  /  =  1 

Step  2.  \N‘hile  i  <  A'o  do  Steps  3  6. 

Step  3.  Set  fi  =  po  -  /(pi))//'(P(i),  {Coiiipiife  p,.) 

Step  4.  If  I  p  -  p„  |<  TOL  then  OUTF'UT  p;  STOP. 

Step  5.  Set  I  =  ;  +  1 . 

Step  6.  Set  p(i  =  p. 

Step  7.  OUTPUT  (‘Method  failed  after  A'o  iterations,  .A’o  =’,.Vo;  STOP. 
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Angus  et  al.  note  a  problem  with  the  Newlon-Raphson  metliod,  and  state 


In  the  actual  use  of  the  Newton- Raphson  method,  convergence  of  the  estimators  to 
finite  values  could  not  always  be  obtained.  The  major  problem  seemed  to  be  in  finding 
successful  starting  points  for  the  parameter  estimates  as  inputs  to  the  program.  In 
general,  no  real  guidelines  were  found  [4:194]. 

As  the  maximum  likelihood  estimation  of  parameters  for  both  models  is  based  on  the  single  pa¬ 
rameter  /?i ,  this  requires  only  one  initial  starting  point  necessary  for  the  Newton-Raphson  method 
[64:.526].  Musa  et  al.  suggest  an  initial  estimate  for  l3\  to  be  t”’,  the  inverse  of  the  total  testing 
time,  and  state  that  “this  value  almost  always  results  in  the  initial  convergence  of  the  Newton- 
Raphson  procedure”  [64:527],  Therefore,  t~^  will  be  u.sed  as  the  initial  estimate  for  the  parameter 

Applying  the  Burden  and  Faires  algorithm  to  equations  4.5  and  4.6  requires  the  first  derivative 
of  each.  Taking  the  first  derivative  of  Equation  4.5,  we  get 


/'(/?! )  =  (m,) 


-  Ip 


(4.7) 


and  taking  the  first  derivative  of  Equation  4.6  gives 


/Vi) 


-t, 


( 1  +  0iU)' 


1 


1  -t-  3i(, 


\  +  ln(l  -b  3^1,)) 

[(1  +  3\f()  hi(l  -t-  /I]tf)]- 

(4.8) 


/,.5.1.2  Addiiioiial  hnlial  Parawder  Ldinialion.  Ecjuation  4.(  then  is  used  to  calcu¬ 
late  an  estimated  //o  =  3o  (or  tl'O  Execution  Time  model  [64:.T25] 


3i<  — 


1  - 


(4.9) 


Recalling  that  3i  =  Xo/i-'o  for  the  Execution  'lime  model,  the  estimated  initial  failure  intensity 
value  Ao  i-s  then  calculated  as  [64:351,529] 


Similarly.  Equation  4.8  is  used  to  calculate  an  estimated  6  ^  =  po  for  the  Logarithmic  Poisson 
Execution  Time  model  [64:326] 


m. 


ln(l  +  P\if) 


(4.11) 


llecalling  that  d\  —  AqI?  for  the  Logarithmic  Poisson  Execution  Time  model,  the  estimated  Aq  value 
can  also  be  calculated  from  Equation  4.10  [64:351], 


Confidence  Intervals.  Confidence  intervals  for  the  estimated  parameters  were 
developed  based  on  the  assumptions  of  a  normal  distribution,  zero  mean,  unit  variance,  and  a 
desired  confidence  interval  of  95  percent  [64:316).  Such  an  approach  allows  a  100(1  —  a)  percent 
confidence  interval  to  be  calculated  for  the  unknow'ii  mean  /j  from  the  sampling  distribution  of  ,V 
the  sample  mean  [38:242],  The  general  form  of  the  equation  for  this  two-sided  confidence  interval 
is  [38:242] 


A'  - 


Zaf2<^ 

x/n 


<  /r  <  A  + 


'o/2<^ 


(4.12) 


For  a  95  percent  confidence  interval  a  =  .05  with  o/2  =  .025.  From  a  cumulative  standard 
normal  distribution  table,  the  test  statistic  A025  =  1-96  [38:243, -593].  Musa  et  al.  apply  this,  as 
well  as  the  unit  variance  assumption  of  tr  =  1,  to  Equation  4.12  and  derive  the  following  version  of 
the  two-sided  confidence  interval  for  the  estimated  parameter  0k  [64:316] 


3k  ± 


^'l-o/2 

/iuh) 


(4.13) 


with  being  “the  appropriate  normal  deviate”  and  /(d* )  being  the  “expected,  or  Fisher, 

information”  [64:315-316].  The  appropriate  normal  deviate  eriuates  to  the  test  statistic 


^1-0/2  —  A.02.T  —  1.96 


and  the  expected  information  for  I{0\)  can  be  determined  for  the  Execuiion  Time  model  from 
[64:351] 

r  I  tV-''  1 

(5; -(;«r:rnp I 

with  the  value  for  the  Logarithmic  Pois.son  Execution  Time  model  determined  from  [64:334] 
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I(0l)  = 

I _ _ 1  [  _  1 

<;[ln(l  + /3i<c)  +  1]  ) 

[(l  +  /?i/,)ln{l+ft<,)p/ 

Equations  4.14  and  4.15  are  then  substituted  into  Equation  4.13  to  determine  tlie  upper  and  lower 
95  percent  confidence  parameters  of  /?] ,  All  three  values  (/?],  and  l3\high  )  are  used  in  Equation 

4.9  to  determine  i/q  and  its  confidence  boundary,  and  also  in  Equation  4.11  to  determine  0  and  its 
confidence  boundary.  The  results  of  these  are  then  used  in  Equation  4.10  to  determine  Ao  and  its 
95  percent  boundary.  The  different  values  of  Aq  and  uq  are  used  in  Equations  2.8,  2.9,  and  2.10 
to  evaluate  the  applicability  of  the  Execution  Time  model,  while  the  different  values  of  Aq  and  0 
are  used  in  Equations  2.11,  2.12,  and  2.13  to  evaluate  the  applicability  of  the  Logarithmic  Poisson 
E.xecution  Time  model. 


4.3.2  Model  Parameter  Dert nation.  Applying  the  techni()ues  and  equations  identified  in  the 
previous  section  to  strictly  DTAjE  data  results  in  a  final  failure  intensity  that  can  be  based  either 
on  time  of  last  failure  (A(r))  or  on  the  number  of  failures  experienced  at  that  time  (A(/i))  [64].  As 
these  valties  are  at  the  end  of  DTA’E,  they  also  represent  the  failure  intensity  values  at  the  start 
of  the  next  phase  of  testing,  fOTAE.  Therefore,  tlie  value  of  A],  is  known  at  the  start  of  lOTA  E. 
,-\ssuming  additional  data  are  not  available  (either  with  respect  to  failures  or  system  characteristics), 
calculation  of  the  initial  parameter  /?]  was  based  on  the  equations  used  to  derive  Aq 

The  equation  for  Aq  for  tlie  E.xecution  Time  model  is  ba.sed  on  Equation  4.10,  and  in  its 
e.xiiandi'd  form  is  [64:351] 


with  the  expanded  form  of  the  Logarithmic  Poi.s.son  Execution  Time  model  also  based  on  Equation 
4.10  and  given  by  [64:351] 


ln(l  +/?i<e) 


(4.17) 


Subtracting  Aq  from  both  sides  and  setting  these  equations  equal  to  0  allowed  the  Newton-Raphson 
method  to  be  used  to  determine  the  value  of  /?i . 
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4.5.2.]  Neu'ton-Raphson  Method.  Again  applying  the  Burden  and  Faires  algorithm, 
the  first  derivative  of  Equation  4.16  is 


/'(A) 


(1-e  ^*‘0 

(1 


and  taking  the  first  derivative  of  Equation  4.17  gives 


m)  = 


(ln(l  +  /?l<e))(»ne)  -  (mePl){ 
(ln(  1  +  P^fe))- 


(4.18) 


(4.19) 


Therefore,  the  initial  derivation  of  P\  was  determined  after  Aq.  While,  these  equations  use  typical 
end-of-test  variables,  such  as  and  in,,  these  variables  are  cumulative  and  can  reflect  even  the 
early  stages  of  testing.  For  the  purpose  of  this  study,  only  final  lOTicE  data  was  used  after  initial 
parameter  derivation  from  DTicE  data,  as  this  was  believed  to  provide  a  better  description  of  the 
mapped  models. 

4. 5. 2. 2  Additional  Initial  Parameter  Derivation.  Once  /?i  was  derived,  other  initial 
values  were  then  derived.  For  the  Execution  Time  model,  i/q  =  Pq  was  derived  from  Equation  4.9, 
while  Equation  4.11  was  used  to  derive  0~^  —  Po. 


4. 5. 2. 3  Confidence  Intervala.  Confidence  intervals  for  the  derived  parameters  were 
developed  based  on  the  assumptions  and  equations  presented  in  the  previous  section  on  model 
parameter  estimation.  Once  boundary  values  were  derived,  those  values  along  with  Aq  and  i/q  were 
used  in  Equations  2.8.  2.9.  and  2.10  to  evaluate  tlie  applicability  of  the  Execution  Time  model, 
and  the  different  values  of  Ao  and  0  were  used  in  Equations  2.11.  2.12,  and  2.13  to  evaluate  the 
a|)plicability  of  the  Logarithmic  I’oisson  Execution  I’inie  model. 


4.6  .4s.se.<i.s  the  Modtl.s 

Implementation  and  code  development  was  conducte<l  in  accordance  with  the  software  devel¬ 
opment  lifecycle,  and  documented  as  such.  A  modular  approach  was  u.sed  with  the  code  to  facilitate 
changes  during  the  experimental  process.  This  proved  useful,  as  an  additional  module  was  added 
during  the  motlels'  evaluation.  The  exact  implementation  details  of  the  analysis  code  are  included 
in  Appendix  D.  An  assessment  of  the  models  and  their  performance  follows  in  the  next  chapter. 
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^.7  Define  and  Implement  Data  Collection  Procedures. 

As  failure  and  date  data  had  already  been  collected,  the  only  additional  effort  was  to  locate 
the  test  duration  and  time  information  needed  for  the  models.  The  results  of  this  are  given  in  the 
following  chapter.  Future  efforts  to  collect  software  reliability  data  must  include  such  test  duration 
and  test  time  as  important  information.  This  also  will  be  discussed  in  the  following  chapters. 

4.S  As.sfss  the  Software  Reliability. 

This  is  the  ne.xt  logical  step,  and  involves  actual  implementation  of  the  candidate  models  on 
a  real  project  with  actual  data.  Such  an  assessment  of  the  software  would  be  based  on  the  models' 
results.  As  the  goal  of  this  thesis  is  to  evaluate  the  software  reliability  models  and  not  the  reliability 
of  the  test  data  software,  comments  concerning  the  reliability  of  the  test  data  software  is  limited 
to  discussion  of  the  models'  applicability  and  not  the  software  systems’  reliability,  f'rom  this,  a 
proposed  lOTfcE  software  reliability  methodology  will  be  discussed  in  the  following  chapters. 

4.9  Summary 

This  chapter  identified  the  implementation  strategy  for  assessing  the  candidate  software  reli¬ 
ability  models.  As  the  integration  of  software  reliability  is  new  to  operational  test  and  evaluation  of 
weapon  systems,  this  chapter  also  identified  the  place  a  software  reliability  model  implementation 
strategy  would  have  in  the  lOTAcE  environment.  Results  and  discussion  of  the  candidate  software 
reliability  models'  implementation  follow  in  the  ne.xt  chapter. 
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I’.  Findings 


This  cliapler  presents  tlie  initial  data  analysis  findings,  tlie  findings  of  the  fitted  models  with 
respect  to  the  actual  failure  data,  a  comparison  of  the  failure  intensity  values  for  each  data  set,  and 
an  evaluation  of  each  model  fitted  for  lOTAcE  failure  data  from  historical  DTAcE  failure  data. 

5.1  Initial  Data  Analysts 

The  basic  data  fields  listed  in  Table  4.3  were  not  sufficient  for  u.se  with  a  software  reliability 
measurement  model,  as  they  were  lacking  .some  sort  of  failure  lime  indication.  .Additional  informa¬ 
tion  on  timing  and  system  characteri.stics  was  identified  (4.5,  47];  however,  of  the  initial  17  data  sets 
available,  only  five  had  sufficient  supplemental  information  to  make  the  maturity  data  meaningful 
in  a  software  reliability  sense.  Therefore,  the  initial  data  analysis  was  conducted  using  only  these 
five  data  sets.  Line  charts  were  plotted  for  each  using  cumulative  total  failures  for  the  j/-a.\is  and 
e.xecution  test  time  for  the  j'-a.\is  to  visually  determine  the  trends  of  each  curve.  The  results  are 
shown  in  Figures  5.1  through  5.6  and  discussed  below. 

5.1.1  Data  Set  Al.  The  test  durations  used  for  this  data  set  varied  from  30  to  738  minutes. 
Although  the  lOTAE  dates  were  from  July  1984  to  June  1989  and  test  durations  and  dales  were 
available  for  the  entire  lOTckrE  period,  the  available  data  from  the  HQ  .AFOTEC  software  maturity 
dalaba.se  (named  SN’^Tf’RR)  'ovep  d  only  the  dates  of  27  .August  1987  to  30  .April  1988,  inclusive, 
with  one  lone  data  point  on  1  October  1980  (see  .Appendix  B)  [15].  This  totaled  six  months  of  data, 
with  a  total  of  14(35  failures  indicated.  The  lone  data  point  was  I'xcluded  from  initial  and  subsequent 
analysis  as  the  test  duration  lime  span  between  this  and  the  next  data  point  was  too  great  for  the 
point  to  be  meaningful.  This  a.ssumption  was  l)a.s«'d  on  the  author's  personal  <“xperience  from 
performing  lOTAE  on  this  weapon  .sy.stem.  .Also,  if  the  trend  is  slat istically  sound,  the  absence  of 
one  data  point  on  either  end  should  not  affect  the  overall  integrity  of  the  data. 

Initial  analysis  of  the  cumulative  failure  data  reveals  an  exponential-like  trend  with  respect 
to  execution  lest  lime  (.see  Figure  5.1 ).  This  is  encouraging,  as  the  software  maturity  data  is  based 
on  calendar  lime  (independent  of  e.xecution  time  or  test  duration),  and  itself  has  an  exponential 
tendency  [94].  .Although  the  exact  time  of  each  failure  occurrence  was  not  known,  times  were 
assumed  to  follow  a  uniform  distribution,  and  were  a.ssigned  randotniy  to  each  failure  event  within 
the  total  test  duration  for  that  calendar  day  [64:1.58]. 

There  was  some  difficulty  mapping  dates  of  failures  to  the  dates  of  actual  test  durations. 
In  some  cases,  dates  listed  for  failures  did  not  have  a  dale  of  test  duration,  and  conversely  some 
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Figure  5.1.  Cumulative  Failures  vs  Execution  Time  for  Data  Set  A1 

test  durations  did  not  have  associated  dates  of  failures.  The  software  listed  in  Appendix  D  was 
modified  to  include  a  special  module  that  would  compensate  for  this  discrepancy  as  follows.  If  lest 
durations  did  not  have  associated  failures  (or  dates  had  multiple  test  durations),  the  test  times  were 
added  to  the  total  test  duration  as  an  offset.  Failures  that  had  no  associated  test  durations  were 
then  added  together  until  an  existing  test  duration  date  was  reached,  and  all  were  applied  against 
that  date.  Admittedly  this  approach  seems  unfair  in  that  failures  listed  between  test  durations 
should  be  applied  against  the  previous  test  duration  (as  that  is  likely  to  be  where  the  failures  were 
found):  however,  given  the  seeming  randomness  in  as.socialion  between  date  of  failure  and  date  of 
test  duration  the  method  used  should  not  unduly  skew  the  data.  The  only  visible  instance  of  this 
smoothing  is  the  rather  flat  slope  directly  in  the  middle  of  the  curve.  Again,  as  the  overall  curve 
tended  towards  exponential,  this  smoothing  should  not  have  any  affect  on  the  data  or  subsequent 
calculations. 

Thus,  it  appears  initially  that  both  the  Execution  Time  Model  and  the  Logarithmic  Execution 
Time  Model  should  fit  this  data  distribution;  however,  as  HQ  AFOTEC  is  involved  with  several 
different  types  of  weapon  systems,  additional  data  sets  must  be  analyzed  for  model  applicability. 
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5.1. '2  Data  Set  A2.  The  lOTAiE  for  this  system  was  from  December  1988  to  September 
1989,  during  wliicli  there  were  512.4  hours  of  testing  with  304  total  testing  periods  [45].  The  failure 
data  available  ranged  from  24  February  1987  through  25  July  1989.  During  the  lOTAcE  timeframe, 
there  were  eight  months  of  testing  and  a  total  of  47  recorded  failures.  .An  initial  assumption  was 
made  that  each  lest  duration  was  1.686  hours  long  (512.4/304=1.686);  however,  there  were  only 
37  failure  dates  listed  from  the  SYSTERR  database  for  the  lOTAcE  period  which  would  leave  267 
test  durations  unaccounted  (see  Appendi.x  B). 

Since  the  number  of  failure  dates  did  not  correspond  in  any  way  to  the  number  of  test 
periods,  another  way  to  determine  the  failure  to  test  duration  relationship  was  needed.  .Available 
information  for  average  test  durations  of  similar  weapon  systems  was  used  as  a  starting  point  to 
determine  an  appro.ximate  relationship.  The  average  number  of  test  flights  per  aircraft  per  month 
for  a  fighter  type  aircraft  is  10  flights/aircraft/month,  with  the  average  number  for  a  larger  type 
aircraft  (such  ;is  a  bomber)  being  5  flights/aircraft/monlh  [1;3], [48:144].  .A  similar  test  program 
used  four  total  aircraft  for  testing  [45].  Therefore,  an  assumption  was  made  that  four  aircraft  were 
used  with  each  having  10  test  flights  per  month.  This  gave  an  appro.ximate  total  of 


(4  aircraft)(10  flight.s/month)(10  mont.bs)=400  sorties  (or  test  durations) 

that  would  have  occurred  from  December  1988  to  September  1989.  .As  the  actual  number  of  test 
dnratiotis  was  less  than  the  estimated  number,  and  assuming  a  standard  normal  distribution,  either 
the  assumed  number  of  test  aircraft  should  be  reduced  giving 


(3  aircraft)(10  fiights/month)(  10  montlis)=300  sorties 
or  the  number  of  flights  per  month  should  be  reduced  giving 


(  I  aircraft  )(8  flights/month)(  10  monlhs)=320  sorih's 

Varying  the  number  of  test  aircraft  yields  the  closer  app.o.ximation.  with  the  additional  time  from 
the  last  four  sorties  easily  applied  to  the  last  month  of  testing  (which  is  acceptable,  as  there  is  no 
failure  data  for  any  month  past  July  1989).  Therefore,  30  test  durations  of  1.686  hours  each  (-50.58 
total  test  hours)  were  assumed  to  occur  each  month,  with  34  test  durations  of  1.686  hours  each 
(57.32  total  test  hours)  assumed  to  occur  in  the  last  month  of  testing. 

Cumulative  total  number  of  failures  were  determined  for  these  durations  ba.st'd  on  the  fol¬ 
lowing.  .Assuming  a  normal  distribution  for  the  dales  of  test,  each  month  was  treated  as  a  total 
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test  clu  ation  of  50.58  hours  (57.32  for  the  final  month).  The  number  of  failures  per  month  uere 
then  added,  and  assigned  randomly  within  that  test  duration.  The  results  are  shown  in  Figure 
5.2.  By  inspection,  the  data  appears  to  follow  some  form  of  exponential  curve.  While  the  trend  is 
more  .9-sliaped,  there  appears  to  be  enough  of  an  exponential  shape  to  proceed  with  the  candidate 
models  on  this  data  set  as  well. 


Failures  (Cumulative) 
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Figure  5.2.  C'umulative  Failures  vs  Execution  Time  for  Data  .Set  .•\2 


■5. /..I  Data  S(t  .'1  :f.  There  were  219  test  periods,  fotir  lest  aircraft,  and  an  average  test 
duration  of  1.5  hours  for  IOT<*k:K)  of  this  system  which  lasted  from  23  May  198!)  to  1  November 
1989  [•1.5].  This  gave  5.25  months  of  lOTAcE  and  -50  recorded  failures.  Using  the  relationship  defined 
above,  that  gives 


(4  aircraft)(10  flight.s/month)(5.25  months)=210  sorties 

which  is  extremely  close  to  the  219  actual  test  flights.  Varying  tlte  number  of  flights  per  month 
(which  is  itself  an  average)  to  11  gives 
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(4  aircraft)(ll  flights/month)(5.25  montlis)=231  sorties 


A  closer  approximation  was  obtained  by  taking  tlie  219  sorties  and  dividing  back  by  the  number 
of  months  (5.25),  which  yields  41.7  test  flights  per  month.  At  1.5  hours  each  (on  the  average)  the 
total  test  time  per  month  is  then 

(41.7  test  flights)(1.5  hours/test  flight)=62.55  hours 

with  the  first  month  of  testing  having  only  15.6  test  hours  due  to  only  8  days  of  testing  occurring 
in  the  first  month. 

Cumulative  total  number  of  failures  were  determined  for  these  test  durations  based  on  the 
same  assumptions  that  were  used  with  the  A2  data  set.  A  normal  distribution  was  assumed  for  the 
dates  of  test,  with  each  mouth  treated  as  having  a  total  test  duration  of  62.55  hours  (15.6  for  the 
first  month).  The  number  of  failures  per  month  were  then  added,  and  assigned  randomly  within 
that  test  duration.  The  results  are  shown  in  Figure  5.3.  This  curve  exhibits  more  dramatic  changes 
in  the  cumulative  failures  than  the  previous  data  sets.  Even  so.  the  general  trend  should  permit 
the  use  of  the  candidate  models. 

5.1.4  Data  Set  Si.  lOTirE  for  this  system  lasted  from  3  February  1988  until  6  July  1989 
[45].  A  total  of  five  two-week  test  periods  occurred  at  three  different  test  sites  (two  two-week 
periods  of  testing  at  one  site,  two  two-week  periods  of  testing  at  another  site,  and  one  two-week 
period  of  testing  at  the  third  site),  with  an  average  of  20  hours  per  day  of  testing  for  14  straight 
days  [47]. 

The  use  of  three  different  test  sites  normally  retpiires  adjusting  the  test  durations  and  times 
of  failure  occurrences.  Musa  et  al.  provides  an  excellent  description  of  how  to  interleave  test  time 
and  failure  occurrence  for  multiple  te^t  installations  [64:162-165].  .Normally,  one  would  think  to  use 
independent  failure  intervals  for  each  program,  as  with  the  hardware  for  a  systt'in;  however,  due  to 
the  logical  nature  of  software  a  failure  and  t<?st  lime  interleaving  is  more  appropriate  [64:162-165]. 

For  this  ap|dication.  the  exact  lime  of  each  failure  occurrence  is  not  known.  Therefor*', 
interleaving  is  not  applicable,  and  it  will  be  sufficient  to  take  the  total  test  duration  of 

(20  hours  testing  per  system  per  day)(3  systems)=60  hours  testing  per  day 

and  divide  that  by  the  number  of  failures  occurring  on  that  day.  Since  the  five  two-week  test 
periods  were  well  within  the  start  and  slop  dale's  for  lOTAE.  and  there  were  failure  data  for  other 
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Figure  5.3.  Cumulative  Failures  vs  Execution  Time  for  Data  Set  A3 

dates  inside  the  IOTi:E  timeframe,  the  total  10T&:E  time  was  considered  to  be  ^  (60  hours  per 
day)(number  of  days  in  the  month)  for  a  total  of  J6  months  of  failure  tiata  (see  .Appendix  B).  The 
exact  test  time,  therefore,  varied  with  the  number  of  days  in  the  month  and  totaled  27780  hours 
(an  average  of  1736.25  test  hours  per  month).  Tliere  were  413  recorded  failures.  The  results  of 
this  are  shown  in  Figure  5.4.  This  data  set  h2ts,  by  far,  exhibited  the  closest  approximation  to  an 
exponential  curve.  Therefore,  the  candidate  models  should  work  very  well  with  this  dataset. 

5.1.5  Data  Sel  \VJ.  There  were  no  lOT&E  dates  nor  lest  durations  given  for  this  system 
[45].  The  total  S^'STERR  database  was  used,  resulting  in  an  a.ssiimed  7  months  of  lOTArE  with 
450  recortled  failures.  Therefore,  based  on  the  author's  limited  involvement  with  a  similar  system 
and  the  freijuency  of  failure  dates,  an  initial  assumption  was  made  that  all  tests  dates  were  valid 
data  points,  test  durations  only  occurred  on  the  dates  of  failure  identification  (as  determined  from 
the  SYSTERR  database),  and  that  each  test  duration  was  six  hours  long.  This  resulted  in  the  data 
increasing  in  a  linear  fashion  (see  Figure  5.5). 

Subsecpient  research  indicated  that  actual  test  durations  were  16  hours  each,  and  another 
assumption  was  made  that  testing  was  conducted  for  five  working  days  each  week  [46],  .At  an 
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Figure  5.4.  Cumulative  Failures  vs  Execution  Time  for  Data  Set  SI 

average  of  22  working  days  per  month  (or  4.4  weeks  per  month),  and  still  using  all  failure  dates, 
that  results  in 


(22  working  days  per  month)(l()  hours  te.stiiig  per  day)=3.52  hours  testing  per  month 

or  SO  hours  of  testing  per  week.  The  .esults  of  this  new  calculation  are  shown  in  Figure  5.6,  and 
the  data  distribution  is  much  more  exponential  than  niuler  the  previous  a.ssumplioiis  This  is  not 
an  instance  where  the  assumptions  were  changed  to  provide  data  that  fit  the  models;  instead,  the 
initial  assumptions  were  modified  as  additional  data  became  available.  Based  on  the  additional 
data,  the  candidate  models  should  also  be  applicable  to  this  data  set. 

5. '2  Calculated  Values  for  Current  Number  of  Failures  Compared  to  Actual  Number  of  Failures 

After  an  initial  model  feasibility  assessment  of  the  data  indicated  the  candidate  models  were 
feasible  for  the  data  sets  based  on  the  data  sets’  apparent  exponential  distributions,  parameter 
estimates  were  obtained,  fitted  models  wer'»  derivinl.  and  goodness-of-fit  tests  performed  for  each 
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Figure  5.5.  Cumulative  Failures  vs  E.xeculion  Time  for  Data  Set  \\T,  Initial 

inociel/dala  set  combination.  It  is  helpful  at  this  point  to  redefine  the  goal  of  this  thesis  in  terms 
of  a  null  hypothesis  such  that  [38:280] 


Ho  .0  =  00 
H,  .0^  Oo 

where  6o  is  a  parameter  being  assessed  against  an  [i.C]  interval  with  100(1  — o)  percent  confidence 
[38:280].  The  test  then  leads  to  rejection  of  the  null  hypothesis  Ho  if  the  parameter  Oo  i?^  outside 
the  05  percent  confi<lence  interval  [38:280]. 

The  second  a.ssessment  is  concerned  with  the  calculated  values  for  “current’’  number  of  failures 
compared  to  the  actual  number  of  failures  for  any  given  time  during  the  entire  lOTicE  test  period. 
Therefore,  0  is  the  actual  number  of  failures  e.xperienced.  and  the  parameter  Oo  is  e.xpected  number 
of  failures  at  time  r,  or  p(r).  derived  from  Equations  2.8  and  2.11.  The  [L,U]  boundaries  are 
calculated  for  the  initial  parameter  l3\  based  on  Equation  -1.13  and  either  Equation  4.14  for  the 
Execution  Time  model  or  Ecpiation  4.15  for  the  Logarithmic  Poisson  Execution  Time  model.  The 
parameter  and  its  bouiularies  are  then  used  in  Equation  2.8  for  the  Execution  Time  model  and 
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I'iguro  5.().  Cumulative  Failures  vs  Execution  Time  for  Data  Set  \V1 

E(iuation  2.11  for  the  Logarithmic  Poisson  Execution  Time  model.  The  results  of  this  are  shown 
in  Figures  o,7  liirough  a. Id  and  discussetl  helow. 


5.2./  Data  S<t  A  I .  The  results  of  the  fitted  Execution  Time  model  application  to  the  data 
are  shown  in  Figure  -5.7.  Etpiation  2.8  was  fitted  with  values  of  the  initial  parameters  Aq  = 
0.1()2871()1  1  and  un  =  1(528. 7-1  to  get  the  following  ei|uat ion  for  failures  expected  at  time  r 


//(-)  =  1(528.71 


—  ex|> 


/  0.1(52871(51  1  \ 
V  1(528.7-4  V 


(.5.1) 


The  actual  data  (ends  outside  the  projected  d~>Vi  confidence  intervals;  however,  this  represents 
only  a  small  part  of  this  weaitons  system’s  entire  IOTA:E  effort.  The  tendency  outside  the  confidence 
intervals  could  be  dtie  to  (he  small  snapshot  of  data  used  (6  nionths  of  recorded  maturity  data 
compared  to  alniost  -5  years  of  lOTCE).  or  to  the  failure  time  assignment  process.  This  process 
prohibits  identifying  exact  failure  times  (there  was  no  initial  correlation  between  maturity  failures 
and  dates  of  testing),  and  results  in  reporting  t  he  failures  as  "lump  sums"  at  varying  time  intervals 
based  on  a  calendar  date  relationship.  Therefore,  while  we  apparently  reject  the  null  hypothesis. 
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additional  test  data  on  either  end  of  the  curve  for  a  substantial  amount  of  time  would  provide  a 
more  accurate  assessment  of  the  Execution  Time  model. 
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Figure  5.7.  Expected  Failures  Using  Execution  Time  Model  for  Data  Set  A1 

This  same  observation  holds  for  the  Logarithmic  Poisson  Time  model,  whose  results  are 
shown  in  Figure  5.8.  Equation  2.11  was  fitted  with  Aq  =  0.322609809  and  $  —  0.001883754  ?is 
initial  parameters  to  get  the  following  equation  for  failures  expected  at  time  r 

ln((0.322609809H0.0018837.54)r+  1)  (5.2) 

The  Logarithmic  Pois.son  Time  model  dirl  provide  a  closer  fit  to  the  data;  however,  three 
of  the  five  data  “lump  sums”  were  significantly  outside  the  confidence  intervals,  and  we  therefore 
reject  the  null  hypothesis.  In  this  ca.se,  a  more  accurate  determination  of  failure  times  could  provide 
a  better  representation  of  failure  times,  and  pos.sibly  an  even  closer  fit  of  this  model. 
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Figure  5.8.  Expected  Failures  Using  Logarithmic  Poisson  Execution  Time  Model  for  Data  Set  AJ 

5.2.2  Data  Set  A2.  The  fitted  form  of  the  Execution  Time  model  had  initial  parameters 
Ao  =  0.003096C31  and  =  73.2-1  to  get  the  following  equation  for  failures  expected  at  time  r 


^t(T}  =  , 


3.24 


/  0.00309(3631  \ 

'-""i — ?3:5r-v 


(5.3) 


There  was  a  substantially  better  fit  of  the  Execution  Time  model  to  the  A2  data  than  the 
-M  data,  as  can  be  seen  in  Figure  5.9.  .-Ml  but  the  two  initial  data  points  were  within  the  95% 
confidence  intervals.  This  trend  is  not  uncommon  for  the  Execution  Time  model,  which  tends  to 
perform  more  satisfactorily  after  the  first  60%.  of  the  test  time  period  [64,  89],  Overall,  there  was 
a  good  fit  of  the  model  to  the  data,  and  we  fail  to  reject  the  null  hypothesis. 

Similarly,  the  Logarithmic  Poi.sson  Execution  Time  model  performed  better  for  this  data  set 
than  for  the  previous  data  set  (see  Figure  5.10).  The  model  was  fitted  with  Aq  =  0.003267847  and 
0  =  0.020626162  as  initial  parameters  to  get  (he  following  equation  for  failures  expected  at  time  r 

=  n  WmT-Tr-TT-i  l"((0  003267847)(0.020.626162)7  +  1)  (5.4) 

0.020()2()1()2 
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Figure  5.9.  Expected  Failures  Using  Execution  Time  Mode)  for  Data  Set  A2 

One  possible  reason  for  the  better  fit  could  be  the  data  set  being  complete  with  respect 
to  the  amount  of  lOTArE  test  time  and  number  of  failures  recorded,  while  the  previous  A1  data 
set  contained  only  a  portion  of  the  overall  operational  testing  effort.  It  is  interesting  to  note  the 
Logarithmic  Poisson  Execution  Time  model  does  not  fit  as  well  to  the  data  as  does  the  Execution 
Time  model.  This  could  be  due  to  failures  not  having  specific  occurrence  times-the  combination 
of  using  average  test  durations  per  tuonth  and  assigning  normally  distributed  random  times  as 
failure  occurrence  times  could  produce  clustering  of  data.  While  these  clustered  points  do  provide 
adequate  trend  analysis,  a  more  accurate  representation  of  the  failure  lime  data  could  indicate  a 
much  closer  model  fit.  As  it  stands,  we  must  reject  the  null  hyiiothesis  for  this  candidate  model 
with  the  data  set. 

5.2.3  Data  Sti  .A3.  The  results  for  data  set  A3  are  similar  to  those  of  data  set  A2,  and  are 
shown  in  Figure  5.1 1.  There  appears  to  be  a  closer  model  fit  for  A3  than  either  of  the  previous  two 
data  sets  using  the  Execution  Time  model,  leading  us  to  fail  to  reject  the  null  hypothesis.  Again, 
this  could  be  due  to  this  data  set  being  complete  with  respect  to  the  data  that  was  available,  even 
though  the  test  times  per  month  were  derived  from  an  average.  The  fitted  form  of  the  Execution 
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Figure  u.lO.  Expected  Failures  Using  Logarithmic  Poisson  Execution  Time  Model  for  Data  Set 
A2 


Time  model  had  initial  parameters  Ao  =  0.005358151  and  t/n  =  82. "d  to  get  llie  following  equation 
for  failures  expected  at  time  t 


fi(r}  =  82.74 


1  —  exp 


(- 


0.005358151  A 
82.74 


(5.5) 


The  actual  numl)er  of  failures  (//)  at  any  given  time  (t)  appears  to  exhibit  an  s-shaped 
tendency.  This  is  also  true  for  the  previous  data  sets  A2  and  Al.  While  this  might  lead  to  the 
conclusion  that  a  model  such  as  the  A'amada-Ohba-Osaki  Power  model  could  be  feasible,  there  is 
another  po.ssible  interpretation.  The  shift  in  the  curve  could  be  due  to  additional  software  relea.se.s 
during  the  lOT&E  time  frame.  Musa  et  al.  present  a  method  of  adjusting  failure  times  for  evolving 
programs  [64:440-448];  however,  the  limited  scope  of  lOT&E  should  not  require  such  adjusting, 
especially  when  the  data  are  located  within  the  confidence  intervals. 

The  Logarithmic  Poisson  Execution  Time  model  also  fit  well  to  the  actual  data  on  failures 
experienced  (see  Figure  5.12).  The  model  was  fitte<l  with  Aq  =  0.005505088  and  0  =  0.017004749 
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Figurf*  5.11.  Expected  Failures  Using  Execution  Time  Model  for  Data  Set  A3 
as  initial  parameters  to  get  tlie  following  equation  for  failures  expected  at  time  t 

f,{r)  =  - r—  •  In((0.00550.5088)(0.0170047-19)7-+  1)  (5.8) 

0.017004749  ' 


■Any  potential  rea.sons  for  tlie  minor  deviations  have  been  previously  discus.sed  for  the  data  sets 
.A  1  and  .•\2.  Overall,  the  model  appears  to  have  a  very  good  fit.  Thus,  we  fail  to  reject  the  null 
hypothesis. 

5.2J,  Data  S<1  Si.  The  Execution  Time  model  had  a  fitted  form  with  initial  parameters 
Xo  =  0.001 17344(5  and  /aj  =  417.03  which  gave  the  following  form  of  the  equation 


(  0.001  17344(3 

„(r)  =  41,.03 


Figure  5.13  shows  the  closeness  of  the  curve  to  the  actual  data,  and  while  the  SI  data  curve 
appears  to  be  steeper  than  the  estimated  curve,  the  fit  is  still  very  clo.se.  One  po.ssible  reason  for  the 
sleeimess  of  the  curve  and  tightness  of  the  95  percent  confidence  intervals  could  lie  the  assumption 
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Figure  5.12.  Expected  Failures  Using  Logaritliniic  Poisson  Execution  Time  Model  for  Data  Set 
A3 

of  uniform  test  time  (tiO  hours  per  <lay)  throughout  the  entire  month.  It  is  possible  that  actual  test 
times  could  flatten  out  the  curve,  resulting  in  a  closer  fit  of  the  model.  Even  so,  we  fail  to  reject 
the  null  hypothesis  due  to  the  closeness  of  the  data  and  actual  curve. 

With  the  exception  of  the  steepness  of  tlie  actual  curve,  the  Logarithmic  Poisson  Execution 
Time  motlel  also  fit  well  to  the  actual  data  (see  Figure  5.1*1);  however,  there  were  enough  ilata 
points  outside  the  confidence  intervals  that  we  reject  tlie  null  hypothesis.  The  Logarithmic  Poisson 
Execution  Time  mo<lel  was  fitted  with  the  initial  parameters  Ay  =  0.001770339aiul  0  —  0. 0071)16991 
to  get  the  following  form  of  the  e<|uation 

/'(')  =  0  007(316991  '"«000>^^0339)(0.007616991)7+  1)  (5.8) 
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Figure  5.13.  E.xpected  Failures  I'sing  Execution  Time  Model  for  Data  Set  SI 


5.2.5  Data  Set  Wl.  The  fitted  form  of  the  Execution  Time  model  had  initial  parameters 
Ao  =  0.001927052  and  i^o  =  —221.00  which  gave  the  following  form  of  the  equation 


fi(r)  =  -221.00 


1  —  exp 


/  0.001927052  \ 

V  -221.00  / 


(5.9) 


This  was  by  far  the  most  interesting  of  the  data  sets  to  analyze.  Figure  5.15  reveals  an  tiicrtasing 
failure  rate.  Musa  et  al.  note  that  if  both  the  initial  parameters  /?o  and  /?]  are  less  than  0.  the 
model  will  exhibit  an  increasing  failure  intensity  [(54:310).  Such  an  indication  does  not  invalidate  the 
model's  application,  since  this  model  is  of  tiie  exponential  Poisson  group  which  “can  accommodate 
increasing  and  decreasing  failure  intensities.”  making  sure  that  /i(f)  and  A(/)  are  both  nonnegative 
[64:310]. 

The  reason  for  this  increasing  failure  intensity  could  be  the  operational  tests  were  designed 
to  e.xercise  the  easier  parts  of  the  system  first,  and  then  the  more  critical  ones  later.  The  rapid 
flattening  towards  the  end  of  testing  would  then  be  indicative  of  a  regression  test  where  only  one 
or  two  new  failures  are  identified.  Still,  the  Execution  Time  model  does  provide  a  fairly  accurate 
mapping  to  the  actual  failure  data  for  the  la.st  half  of  the  test  time.  This  concurs  with  other 
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Figure  5.14.  Expected  Failures  Using  Logarithmic  Poisson  Execution  Time  Model  for  Data  Set 
SI 

oKservations  of  applications  of  this  model  [64,  89].  As  the  test  for  the  null  hypothesis  is  regardless 
of  an  increasing  or  decreasing  failure  inlensily.  we  fail  to  reject  the  null  hypothesis. 

The  results  of  the  Logarithmic  Poi.sson  Execution  Time  model  are  a  little  more  dramatic. 
As  shown  in  Figure  5.16,  the  curve  has  a  very  steep  incline  and  then  a  drastic  flattening.  This 
could  be  ba.sed  on  the  fact  that  this  data  set  has  an  increasing  failure  intensity,  and  although 
geometric  Poisson  group  models  can  “accommodate  decre^Lsing  and  a  certain  type  of  increasing 
failure  intensities,"  (he  initial  model  parameter  /?|  still  diverges  [64:312].  Indeed.  The  Logarithmic 
Poisson  Execution  Time  model  was  fitted  with  the  initial  parameters  Ao  =  73.957034969  (which 
indicates  divergence  in  the  Newion-Raphson  estimation  method)  and  0  =  0.046388798.  resulting 
in  the  following  form  of  the  equation 

p(r)  =  - ? - ln((73.957034969)(0.046388798)T+  1)  (5.10) 

0.046388798 


The  level  of  initial  parameter  divergence  appears  to  affect  the  slope  of  the  curve  in  a  pro¬ 
portional  way.  One  possible  way  to  reduce  the  steep  slope  is  to  test  the  more  failure- likely  areas 
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Failures  (Cumulative) 


Execution  Time  (Thousands  of  Minutes) 

mu  — H—  rTUj(tau) 

Upper  Bour>d  '-'€3'—  Lower  Bound 


Figure  5,15.  Expected  Failures  Using  Execution  Time  Model  for  Data  Set  WT 

first,  before  checking  the  least-likely  failure  areas  of  the  software.  With  the  calculated  data  clearly 
differing  from  the  actual  data,  we  reject  the  null  hypothesis. 

5.3  AsstssmenI  of  Fatlurt  lui^nsity  Values 

The  previous  two  assessments  established  the  models’  feasibility  with  respect  to  the  initial 
data,  as  well  as  the  ''fit”  of  the  model  based  on  parameter  derivation.  This  section  addresses  the 
failure  intensity  calculations  of  both  models. 

The  initial  failure  intensity  (Ay)  and  final  failure  intensities  for  each  data  .set  are  shown  in 
Table  5-1.  The  final  failure  intensity  values  are  listed  for  both  lime  (A(r)^  from  Equations  2.9 
and  2.12)  and  failures  experienced  (A(^)y  from  Equations  2.10  and  2.13).  The  values  for  data  set 
W1  are  very  much  skewed  based  on  the  increasing  failure  intensity  characteristic  of  the  data,  and 
provide  no  insight  into  any  relationship  between  the  failure  intensities.  Data  set  .M  does  not  cover 
its  final  IOTAjE  te.sting  time.  Therefore,  the  final  failure  intensities  can  not  be  used  to  determine 
any  operational  reliability;  however,  it  is  interesting  to  note  the  closeness  of  values  between  the 
two  different  models’  final  failure  intensity  calculations.  While  there  is  considerable  disagreement 
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Failures  (Cumulative) 


Execution  Time  (Thousands  of  Minutes) 

mu  — i —  mu(tau) 

Upper  Bound  D  Lower  Bound 


Figure  5.1(3.  Expected  Failures  Using  Logarithmic  Pois.son  Execution  Time  Model  for  Data  Set 

Wl 

between  tlie  Execution  Time  model  and  the  Logarithmic  Poi-sson  Execution  Time  model  concerning 
the  final  failure  intensities  for  the  test  period,  the  basis  of  calculation  (time  vs.  experienced  failures) 
does  not  seem  to  im|)act  the  specific  calculations  for  each  model. 

.Similarly,  data  sets  A2  and  .^3  have  final  failure  intensity  values  for  each  model  that  are 
relatively  close  to  each  other  regardless  of  calculation  basis  (i.e.-A(r)^  %  X(fj)f).  There  is  a 
substantial  differenc*'  between  the  two  candidate  models  for  each  data  set.  Within  each  data  set 
the  models  yield  clos<'  results  regardless  of  the  input  |)arameter  (time  or  failures). 

Data  set  Si  seems  to  exhibit  a  more  ideal  failure  intensity  trend.  The  values  for  each  model 
are  almost  identical  regardless  of  the  input  parameter  (time  or  failures)  and  appear  to  decrease  to 
a  more  favorable  level.  Taking  one  of  the  final  failure  intensities,  such  as  A(r);  =  0.000011327  for 
the  Execution  Time  model,  the  operating  assumption  can  be  extended  to 


(60  hours  per  day)(365  days  per  year)  =  21900  hours  of  operations  per  year 
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Table  5.1.  Comparison  of  Software  Reliability  Failure  Intensities 


Data  Set 
and  Model 

Initial  Failure 
Intensity  Aq 

A1  (Exec) 

A1  (Log) 

0.162871611 

0.322609809 

0.010574044 

0.018310707 

0.010573845 

0.018310712 

A2  (Exec) 

A2  (Log) 

0.003096631 

0.003267847 

0.001138657 

0.001259321 

0.001109443 

0.001239492 

A3  (Exec) 

A3  (Log) 

0.005358151 

0.00550.5088 

0.002120178 

0.002352398 

0.002120206 

0.002352398 

BHSH 

0.001173446 

0.001770339 

0.000011327 

0.000076181 

0.000011340 

0.000076181 

WT  (Exec) 

W1  (Log) 

0.001927052 

73.957034969 

0.005850893 

0.000169250 

0.005850914 

0.000000064 

which  can  then  be  applied  to  Equation  4.1  to  give  a  reliability  assessment  of 


/?(21900)  =  ^-(0.00001 1327)(21900) 

=  0.780312109 


5-4  Calculaled  Vahies  for  Cumnt  Number  of  Failures  (Based  on  DT&E  Data)  Compared  to  Actual 

Number  of  Failures 

A  fourth  model  feasibility  assessment  was  made  of  the  candidate  models  based  on  the  available 
DT\:E  data.  Parameter  estimates  were  obtained  and  fitted  models  were  derived  for  DTAtE,  from 
which  the  final  failure  intensity  values  were  determined.  These  values  then  served  as  initial  inputs 
to  the  models,  and  another  evaluatio::  similar  to  the  second  cissessment  was  conducted.  The  same 
null  hypothesis  criteria  aiul  goals  apply,  only  the  data  set  has  been  expanded  to  provide  more 
realistic  values  of  the  initial  parameters.  Only  data  sets  A2,  A3  and  SI  had  identifiable  D1\AE 
failures  as  well  as  some  measure  of  test  durations  for  the  DTA;E  timeframe.  The  results  are  given 
below,  and  shown  in  Figures  5.17  through  5.22. 


5J,.J  Data  Set  A'2.  In  order  to  determine  the  final  DT&:E  failure  intensities,  the  assumption 
was  made  that  DTtE  had  the  same  test  times  per  month  as  lOTAcE  (50.58  hrs).  The  final  DTi^E 
failure  intensities  for  both  A(r)  and  A(//)  of  the  Execution  Time  model  were  identical,  providing 
the  lOT^E  initial  parameter  Aq  =  0.001687372.  From  this,  the  fitted  model  was  derived  as 


tt(~) 


-125.65 


/  0.001687372  \ 
V  -125.65  V 


(5.11) 
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The  data  were,  for  the  most  part,  within  the  95  percent  confidence  intervals  (see  Figure  5.17). 
Tlie  interesting  shape  of  this  curve  could  be  due  to  the  initial  Aq  value  derived  from  the  DTAcE 
data.  The  resulting  negative  value  for  hq  is  an  indication  of  an  increasing  failure  intensity.  Since 
the  first  two  assessments  demonstrated  data  set  A2  as  having  a  decreasing  failure  intensity,  the  only 
conclusion  is  the  curve  is  affected  by  the  initial  Aq  parameter.  This,  in  turn,  could  be  a  function  of 
the  assumptions  used  to  determine  the  test  times  for  the  DTi:E  assessment.  Thus,  while  there  was 
a  good  fit  of  the  model  to  the  data,  the  shape  of  the  curve  makes  the  initial  parameters  suspect; 
however,  we  still  fail  to  reject  the  null  hypothesis  based  oti  the  coverage  the  model  provided. 


Failures  (Cumulative) 


mu  — ' —  mu<tau) 

Upper  Bound  ~  C-  Lower  Bound 


Figure  5.17.  E.xpected  Failure.s  Using  Execution  Time  Model  for  Data  Set  A2 

The  Logarithmic  Poisson  Execution  Time  model  also  exhibited  an  increasing  failure  intensity 
trend  (see  Figure  5.18).  The  initial  DTArE  failure  intensity  estimate  was  An  =  15.50*l-l2t)2G(5. 
indicating  divergence.  Therefore,  the  model  was  not  able  to  calculate  a  final  value  of  either  A(r) 
or  A(p)  for  DT&rE.  Instead,  the  lOTicE  model  was  fitted  with  same  initial  failure  intensity  as  the 
Execution  Time  model;  Ao  =  0.001687372.  The  corresponding  B  =  —0.007139135  w’as  derived,  and 
the  equation  for  failures  expected  at  time  t  was 

=  o  bn-.-,Q,-r-  ■  ln((0.001»>87372)(-0.007139135)r+  1)  (5.12) 

— u.uui  loyioo 


.V2I 


Again,  the  same  factors  that  affected  tlie  Execution  Time  model  could  also  have  affected  the 
Logarithmic  Poisson  Execution  Time  model,  especially  since  both  models  used  the  same  initial  Aq 
parameter;  however,  in  this  case  the  model  does  not  fit  the  data,  and  we  reject  the  null  hypothesis. 
Accurate  values  for  test  times  and  failure  times  of  occurrence  could  indicate  a  much  closer  fit  of 
model  and  data. 


Failures  (Gumutative) 


mu 


mu(tau) 


Upper  Bound 


Lower  Bound 


Figure  5.18.  Expected  Failures  Using  Logarithmic  Poisson  Execution  Time  Model  for  Data  Set 
A  2 


5.^.2  Data  Sfl  A  '?.  The  DTAcE  version  of  the  filteil  model  had  Aq  =  0.0021 3()()(>9.  and 
the  resultant  fitted  form  of  the  Execution  Time  model  for  lOTfcE  data  had  initial  parameters 
Ao  =  0.0057291(31  and  i/q  =  75. -12.  This  gave  the  following  erpiation  for  failures  expected  at  time  r 


Ii{t)  =  75.-12 


1  —  exp 


/  0.005729161  Y 

V  <5  .-12  J, 


(5.13) 


Data  set  A3  had  a  closer  fit  of  the  Execution  Time  model  to  data  than  did  data  set  A2  (see 
Figure  5.19).  Tlie  results  were  very  similar  to  those  shown  in  Figure  5.11.  This  closeness  could  be 
due  to  a  closer  approximation  of  DTiUE  final  failure  intensity  values  ba.sed  on  a  better  test  time 
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approximation  (even  though  the  time  used  wtis  an  average).  Therefore,  the  model  maps  well  to  the 
failure  data,  and  we  fail  to  reject  the  null  hypothesis. 


Failures  (Cumulative) 


mu(tau) 


— •  Upper  6oun<l 


Uower  GtounP 


Figure  .').19.  F!xt>ected  Failures  Using  Execution  Time  Model  for  Data  Set  A'.i 

The  Fogarithmic  I’oisson  Execution  Time  model  was  able  to  calculate  a  final  D1\UF,  A  value 
for  both  tim«'  and  failure.s  experienced.  Both  numbers  were  identical,  with  a  value  of  0.000 1 7374 1 : 
howevf’r.  when  ihi.s  number  was  used  as  the  initial  parameter  estimate  for  the  lOTiUE  data,  the 
software  encountered  a  math  overflow  due  to  the  ratio  of  the  small  initial  value  compared  to  the 
lO'FiUE  data  .set.  Thus,  the  lO'l'ArE  initial  failure  intensity  parameter  wa.s  taken  from  DTA:E 
final  failure  intensity  calculations  for  the  Exi'cution  Time  model.  '1  he  initial  parameter  was  then 
Ao  =  0.00572!)  1  ()  1 ,  from  which  0  =  0.01S3977-5!)  was  calculated.  This  gave  the  following  e<|uation 
for  failures  expi‘cted  at  time  r 

^l(r)  =  - ! - ln((0.005729161)(0.018397759)r+  1)  (5.14) 

0.018397759  '  '  '  ' 


The  results  are  shown  in  Figure  5.20.  and  appear  to  be  identical  to  the  second  assessment 
(see  Figure  5.12).  The  model  fit  is  sudiciently  close  that  we  fail  to  reject  the  null  hypothesis.  The 


closeiK-ss  of  botli  the  Execution  Time  model  and  the  Logarithmic  Poisson  Execution  Time  model 
indicate  that  using  DTfcE  data  to  derive  the  initial  parameters  could  be  a  feasible  method. 


Failures  (Cumulative) 


mu 


nHi(tau) 


Upper  Bound 


Lower  Bound 


Figure  5.20.  Ex|)ected  Failures  Usii\g  Logarithmic  f^oisson  Execution  Time  Model  for  Data  Set 
Ad 


5.4  :3  Data  Sci  Si.  The  Execution  Time  model  used  the  DTA'E  final  failure  intensity  value 
Af)  =  0.001850621  as  the  initial  parameter  to  calculate  i/q  =  113.26  and  give  the  following  form  of 
the  equation 


/((r)  =  -113.26 


1  -  exp 


/  0.0018.50621 
V  -113.26  / 


(5.15) 


In  contrast  to  Figure  5.13,  Figure  5.21  shows  the  data  lagging  behind  the  model  throughout 
t  he  ent  ire  IOTA’  E  period.  The  uniformity  of  the  models  curve  and  closeness  of  t  he  estimated  and  95 
percent  confidence  values  could  be  due  to  the  assumption  of  uniform  test  time  during  each  month 
of  testing.  As  with  the  second  assessment,  it  is  possible  that  actual  test  times  could  flatten  out  the 
curve,  resulting  in  a  closer  fit  of  the  model;  however,  as  it  stands  now  there  is  no  fit  between  the 
model  and  the  data,  and  we  reject  the  null  hypothesis. 

As  the  initial  failure  intensity  calculation  for  the  DTA'E  version  of  the  Logarithmic  Poi.sson 
Execution  Time  model  diverged,  the  IOTA.:E  version  was  fitted  with  the  initial  parameter  Aq  = 


Failures  (Cumulative) 


Execution  Time  (Thousands  of  Minutes) 

mu  — > —  mu(tau) 

Upper  BounP  Cj  Lower  Bound 


Figure  5.21.  Expected  Failures  Using  Execution  'Fimc  Model  for  Data  Set  SI 
0.001850621,  from  which  0  =  0.00776-1353  was  derived  to  get  the  following  form  of  the  e(|uation 

^|(T)  =  - ^  •  ln((0.001850621)(0.00776-13.53)r  +  1)  (5.16) 

0.00 1  76d3.)3 

lire  Logarithmic  Poisson  Execution  Time  model  had  a  closer  fit  to  the  data  then  did  the 
E.xeciiiinn  Time  motlel  (see  Figure  5.22).  1  his  trenti  is  very  similar  to  the  one  found  during  the 
second  assessment  (see  Figure  5. 1-1).  Whileihe  model  does  somewhat  approximate  the  actual  tiata. 
there  is  a  sufficient  numher  of  data  points  outside  the  95  pi-rcent  confidence  interval  to  reject  the 
null  hypothesis. 

5.5  .Summary 

1  his  chapter  presented  an  initial  assessment  of  the  applicability  of  each  candidate  model  to 
the  available  data  sets.  Next,  the  results  of  the  fitted  models  were  discussed,  with  a  comparison 
made  between  actual  and  estimated  failures.  The  failure  intensity  values  .'es  were  assessed 

for  any  sort  of  trend  data.  Finally,  for  those  sets  with  sufficient  failure  and  time  data,  the  models 
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Figure  0.22.  Expected  Failures  Using  Logarithmic  Poisson  Execution  Time  Model  for  Data  Set 
SI 

were  run  on  DTtUE  data  to  deternrne  the  initial  parameters  for  I0Ti:E,  and  a  fourth  assessment 
was  |>erfonned  to  see  if  the  models  were  affected  by  previously  existing  failure  data.  The  next 
chapter  contains  conclusions  and  recommendations  concerning  these  evaluations. 
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VI.  Conclusions  and  Recommendations 


Current  operational  test  and  evaluation  of  weapon  system  software  by  HQ  AFOTEC  pri¬ 
marily  empliEisizes  the  operational  suitability  of  the  software.  There  is  no  current  mecisure  of  the 
operational  effectiveness  of  the  software.  In  order  to  provide  some  assessment  of  a  weapons  system’s 
software,  this  thesis  proposed  that  a  software  reliability  model  could  provide  the  needed  level  of 
operational  effectiveness  assessment. 

Only  existing  software  reliability  models  were  considered-no  new  models  were  proposed.  A 
hierarchy  of  software  reliability  models  was  defined,  with  emphasis  on  product  vs.  process  models. 
Within  this  overall  grouping,  four  categories  of  software  reliability  models  were  identified: 

•  Fault  seeding 

•  Input  domain 

•  Times-between-failures 

•  Failure  count 

Software  reliability  model  evaluation  criteria  were  established  that  included: 

•  Predictive  validity 

•  ('apability 

•  Quality  of  .Assumptions 

•  -Apiilicability  to  the  Finite-Time  Environment 

•  Diversity  and  .Applicability  of  Output 

•  Caiiability  to  Use  E.xisting  Data 

Potential  software  reliability  models  from  the  four  categories  were  evaluated  against  these 
criteria.  .A  final  selection  was  made  of  two  candidate  models:  the  Musa  E.xecution  Time  model, 
and  the  Musa-Okiimoto  Logarithmic  Poisson  E.xecution  Time  model. 

Implementation  of  the  candidate  models  wa.s  performed,  and  five  test  data  sets  were  run  to 
assess  t  ne  models’  fit  and  applicability.  Analysis  was  conducted  both  on  the  initial  test  data  sets  and 
calculated  values  for  number  of  failures  and  confidence  intervals.  An  analysis  was  also  performed 
on  the  calculated  failure  intensity  values.  Finally,  three  test  data  sets  were  run  on  historical  DT&:E 
data  to  determine  initial  parameter  estimates,  which  were  then  usetl  for  OTAcE  assessment  of  the 
models'  fit  and  applicability. 
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6.1  Conclustovs 


The  summary  results  of  tlie  null  hypothesis  test  for  each  candidate  model  are  shown  in  Table 
6.1.  There  is  a  generally  good  mapping  of  the  Execution  Time  model  to  the  actual  failure  data,  while 
the  Logarithmic  Poisson  Execution  Time  model  did  not  map  as  well.  The  deviations  outside  the  95 
percent  confidence  intervals  could  be  attributed  to  the  manner  in  which  unknown  time  parameters 
were  estimated  for  the  failure  data.  While  the  data  (failure  and  times)  were  not  exactly  accurate 
and  complete  on  all  accounts,  this  variation  did  give  a  chance  to  evaluate  both  candidate  models' 
robustness  with  respect  to  missing  or  incomplete  data.  With  both  models,  there  was  sufficient 
parameter  estimat  ion  available  to  compensate  for  the  lack  of  exact  failure  and  time  data;  however, 
the  lack  of  data  appears  to  have  a  .significant  impact  on  the  Logarithmic  Poisson  Execution  Time 
model. 


Table  6.1.  Summary  Analysis  of  Hq  Test 


Data  Set 

Musa  Execution 
Time  Model 

Musa-Okumoto 
Log  Model 

A1 

Reject 

Reject 

A  2 

Fail  to  Reject 

Reject 

A3 

Fail  to  Reject 

fail  to  Reject 

SI 

Fail  to  Reject 

Reject 

W1 

Fail  to  Reject 

Reject 

There  is  nothing  definitive  that  can  be  concluded  from  the  comparison  of  failure  intensity 
values.  I’ossibly,  after  gathering  etiough  information  from  different  weapon  systems,  it  might  be 
possible  to  identify  a  trend  in  reduction  of  the  failure  intensities  from  start  of  lOTicE  to  end  of 
lOTA'E,  or  it  might  he  possible  to  idetitify  target  values  for  fitial  failure  intensity  based  solely  on  the 
category  and  type  of  weapoti  system  (e.g. -fighter  aircraft  could  have  the  same  operational  profile, 
and,  therefore,  fly  roughly  the  satne  number  of  hours  per  sortie  or  per  year).  Another  potential 
application  is  in  determining  release  lime  for  the  software;  however,  that  requires  prediction  of  the 
software's  reliability,  ami  is  left  to  future  research  for  validation. 

The  two  previous  analy.ses  were  preliminary,  and  led  to  the  final  assessment  of  using  DTAcE 
data  as  the  basis  for  parameter  estimation,  which  was  then  used  with  the  models  on  lOTAE  data. 
The  results  of  this  assessment  are  shown  in  Table  6.2.  Again,  the  Execution  Time  model  appears 
to  perform  better  than  the  Logarithmic  Poisson  Execution  Time  model;  however,  on  data  set 
A3  where  the  execution  time  data  was  more  accurate,  both  models  performed  well.  This  could 
be  due  to  the  use  of  the  Execution  Time  model  DT&E  final  failure  intensity  value  Afr)/  as  the 
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Logaritlimic  Poisson  Execution  Time  model  lOTAiE  initial  failure  intensity  parameter  Ao  Another 
possible  explanation  is  the  execution  time  data  available  being  more  complete  than  time  data  for 
the  other  data  sets.  A  combination  of  the  two  is  also  possible.  The  closeness  of  the  fit  does  indicate 
the  merit  of  using  DTAiE  maturity  data  as  the  basis  for  parameter  estimation  of  the  models  for 
lOTicE  reliability  measurement;  however,  additional  analysis  with  complete  test  data  is  necessary 
to  state  this  conclusively. 


Table  6.2.  Summary  Analysis  of  //q  Test  for  Data  Sets  With  DTikE  Based  Initial  Parameters 


Data  Set 

Musa  Execution 
Time  Model 

Miisa-Okuiuoto 
Log  Model 

A2 

Fail  to  Reject 

Reject 

A3 

Fail  to  Reject 

Fail  to  Reject 

SI 

Reject 

Reject 

An  extra  evaluation  criterion  discussed  by  Mr  Siefert  in  the  recent  American  Society  for  Qval- 
tly  Control  1st  International  Conference  on  Software  Quality  was  a  more  subjective  assessment  of  a 
software  reliability  model,  namely  “is  it  good”  [79j.  The  candidate  models  presented  in  this  thesis 
exhibit  a  definite  “goodness”  about  them,  which  stems  from  their  straightforward  implementation 
as  well  as  capability  to  use  existing  initial  failure  intensity  data  or  derive  this  information  from 
system  characteristics.  These  capabilities  were  not  found  in  any  of  the  other  models. 

6.2  Recomniendaltons 

There  are  three  other  aspects  of  lOTAoE  software  reliability  models  that  should  be  investi¬ 
gated:  data  needed  for  software  reliability  evaluation;  additional  analysis  of  the  candidate  moilels; 
and  applicability  of  software  reliability.  Thes<'  are  dt.*scribed  in  the  following  sections. 

6.2.1  Data  Seeded  for  Software  Reliability  Evaluation.  The  most  important  aspect  of  soft¬ 
ware  reliability  models  that  appeared  throughout  the  literature  was  that  of  collecting  enough  accu¬ 
rate  and  complete  data.  Unfortunately,  the  <lata  sets  used  for  this  study  were  not  that  accurate  nor 
complete.  The  .4FOTECP  800-2  V'ol  6,  Software  Maturity  Evaluation  Guide,  does  include  a  field 
for  total  operating  time  in  minutes,  which  is  the  time  of  failure  from  the  very  beginning  of  lOTiUE 
[23,  46].  While  such  a  measure  is  good  to  have  (time  of  failure  is  needed),  multiple  testing  that 
can  occur  with  weapon  systems  such  as  aircraft  require  a  simpler  approach  to  collecting  test  and 
failure  times.  One  way  to  simplify  this  is  to  require  tracking  test  duration  (or  test  start  and  »iop 
times),  as  well  as  local  time  of  failure  (failure  time  with  respect  to  that  test,  e.g. -failure  I  occurs 


Table  6.3.  Proposed  Software  Maturity  Data 


Description 

Variable  Name 

Format 

•Software  Problem  Number 

PROB.NUM 

Character  10 

Software  Configuration  Item 

CPCI 

Character  10 

Severity  of  Problem 

SEV.CODE 

Character  1 

Date  Problem  Discovered 

DATE 

Date 

Date  Problem  Fi.xed 

DATE 

Date 

Description  of  Problem 

TITLE 

Character  42 

Test  Identification  Number 

TESTJD 

Character  10 

Date  Test  Planned 

TESTPLAN 

Date 

Date  Test  Completed 

TESTCOMP 

Date 

Start  Time  (minutes) 

START-TIME 

Character  10 

Finish  (End)  Time  (minutes) 

END-TIME 

Character  10 

'l  ime  of  Failure  Occurrence 

TIME-OCCUR 

Character  10 

at  +2.00  minutes).  Tlie  software  model  implementation  can  then  calculate  cumulative  test  times, 
cumulative  failure  times,  and  any  other  needed  statistics. 

In  general,  the  data  necessary  to  applying  software  reliability  models  to  lOTirE  would  include 
the  current  software  maturity  fields,  with  the  exception  of  replacing  the  one  field  for  total  operating 
time  with  the  specific  time  fields  described  above  (see  Table  6.3). 

In  support  of  data  persistence,  an  object-oriented  database  (OODB)  should  be  implemented; 
however,  duo  to  the  newness  and  complexity  ofOODBs.  a  transitional  approach  is  acceptable  where 
the  database  is  described  by  an  object-based  semantic  data  model  (SD.M)  and  then  transformed 
into  an  entity-relationship  diagram  (ERD)  for  implementation  at  the  physical  level.  This  imple¬ 
mentation  ran  then  be  carried  out  with  an  existing  relational  database  model,  such  as  the  one  used 
by  Clipper,  with  virtually  no  loss  to  the  data  meaning  or  relationships. 

Th<’  complete  description  of  these  models  and  their  interrelationships  is  given  in  .Appendix 
E.  along  with  the  SDM  description  for  aircraft  reliability  data.  This  description  can  easily  be 
('xpanded  into  a  superclass  that  would  i;. elude  aircraft,  radar,  missiles,  and  any  other  categories  of 
weapon  systems.  The  SDM  description  includes  not  only  the  failure  data  needed  for  the  weapon 
system,  but  also  the  data  that  will  be  calculated  by  the  software  reliability  models.  From  this,  the 
entity-relationship  (E-R)  diagram  shown  in  Figure  6.1  was  derived.  This  diagram  would  then  be 
the  basis  for  implementing  the  relational  model  to  track  software  reliability. 

Additional  Analysts  of  the  Candidate  Models.  Additional  analysis  of  the  candidate 
models  is  needed  in  the  following  areas:  additional  different  weapon  systems;  use  of  system  char- 


Tail  Number 


Figure  6.1.  E-R  Diagram  for  Software  Reliability  Database 
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acteristics  to  determine  initial  parameters;  evaluation  of  model  adequacy  btised  on  goodness-of-fit 
tests;  impact  of  failure  classification  and  weighting;  sources  of  additional  test  time. 

Additional  Different  Weapon  Systems.  First,  as  sufficient  data  is  accumulated  on 
different  weapon  systems,  the  same  tests  performed  in  this  thesis  should  be  applied  to  see  if  there 
is  agreement  on  the  results. 

Use  of  System  Characteristics  to  Determine  Initial  Parameters.  Next,  the  ca¬ 
pability  to  use  system  characteristics  instead  of  failure  data  to  determine  initial  parameter  values 
should  also  be  done  and  compared  to  the  results  of  the  other  tests.  If  there  is  a  high  correlation 
between  the  three  model  implementations  (using  parameters  determined  from  actual  lOTitrL'  fail¬ 
ure  data,  parameters  determined  from  previous  DTA:E  failure  data,  and  parameters  derived  from 
system  characteristics),  then  the  models  should  be  implemented  for  all  lOTiL’E  test  teams.  The 
viability  of  the  Musa-Okumoto  Logarithmic  Poisson  Execution  Time  model  has  already  been  es¬ 
tablished  by  an  American  Institute  of  Astronautics  and  Aeronautics  (AIAA)  independent  study. 
The  study  was  conducted  by  a  special  “Blue  Ribbon  Panel”  consisting  of  such  software  reliability 
professionals  as  Dr  Farr,  Dr  Hecht,  Mr  Musa,  Dr  Shooman,  Mr  Siefert,  and  others.  The  AIAA 
panel  identified  the  Musa-Okumoto  Logarithmic  Poisson  Execution  Time  model  as  the  best  software 
reliability  model  in  the  time  domain  category,  non-exponential  class  [80:186]. 

Evaluation  of  Model  Adequacy  Based  on  Goodness-of-Fit  Tests.  Finally,  a.^^  data 
on  failure  counts  per  time  interval  becomes  more  thorough,  it  will  be  possible  to  group  the  failure 
data  by  number  of  sample  observations.  Trends  could  emerge  that  would  provide  an  indication  of 
the  expected  number  of  observations  for  time  intervals  throughout  lOTicE.  This  would  then  allow 
and  other  goodness-of-fit  tests  to  be  used  to  test  the  candidate  models’  adequacy  [95]. 

Impacl  of  Failure  Classification  and  Weighting.  This  study  did  not  progress  to 
the  point  of  analyzing  the  individual  categories  of  software  failures.  Research  should  continue 
in  that  direction  to  see  if  there  is  some  relationship  between  the  severity  of  the  failure  and  the 
cumulative  U'st  time.  Also,  potential  acceptability  thresholds  could  be  established  that  allow  some 
categories  of  failuri's  while  requiring  others  to  be  corrected  prior  to  the  end  of  lOT.AE. 

Sources  of  Additional  Test  Time.  Should  the  Test  Director  begin  to  run  out  of 
test  time  before  reaching  his/her  desired  failure  intensity,  alternative  methods  of  testing  might 
increase  the  test  tinre.  For  example,  Adolph  and  Montgomery  identified  the  Integration  Facility 
for  Avionics  System  Test  (IFAST),  which  was  es-sentially  a  hot-mock-up  of  some  of  the  aircraft 


undergoing  test  and  evaluation  at  Edwards  AFB,  CA  [1].  The  use  of  the  IFAST  facility  provided 
additional  test  time,  without  requiring  additional  sorties  from  the  aircraft  and  crew.  An  installation 
such  as  this  would  be  included  as  a  multiple  installation,  and  the  additional  test  time  could  help 
reduce  the  failure  intensity  without  creating  additional  operational  lest  costs.  Methods  of  including 
such  additional  test  time  and  data  should  be  considered  for  integration  into  the  model  databases. 

6.2.3  Applicabiltiy  of  Software  Reliability.  The  candidate  software  reliability  models  can 
potentially  be  used  together  with  system  capability  assessments,  combined  with  hardware  reliability 
models,  applied  to  theoretical  hardware  designs,  integrated  with  other  software  reliability  models, 
or  applied  to  cost  estimation. 

Software  System  Effecttvevess.  Software  reliability  provides  one  way  of  measur¬ 
ing  the  operational  effectiveness  of  the  weapon  system  software;  however,  a  measure  of  the  impact 
of  software  reliability  on  the  total  weapon  system  effectiveness  could  be  determined  as  follows.  The 
ratio  of  software  up-time  to  total  software  “mission”  or  test  time  would  be  determined,  and  this 
value  would  be  the  Software  System  Effectiveness  (SSE)  [t>].  This  number  would  then  be  mul¬ 
tiplied  against  the  desired  Mission  Capable  (MC)  rate  of  the  overall  weapon  system,  giving  the 
Total  Weapon  System  Effectiveness  (TWSE)  [8].  This  result  is  actually  an  adjusted  MC  rate  that 
takes  into  account  the  current  effectiveness  of  the  software.  In  support  of  this,  software  failure 
data  that  indicates  mean-time-torecover  software  (MTTRS)  should  also  be  collected,  and  could 
he  included  as  an  additional  field  of  either  UP.TIME  (time  the  software  was  available  during  the 
test)  or  DOWN_TlME  (time  the  software  was  not  available  during  the  test). 

Combined  Hardware  and  Software  Reliability  Models.  The  concept  of  SSE  was 
somewhat  suggested  in  [42]  as  part  of  a  combined  hardware/software  reliability  model.  A  combined 
model  must  consider  such  “random  phenomena’'  as  the  “software  ‘repair’  process"  where  the  system 
is  restored  "to  an  operational  state  without  correcting  the  .software  fault"  [42:1-1].  Therefore,  even 
if  a  combined  model  is  not  available  in  the  near  future.  MTTRS  and  SSE  data  should  be  collected 
and  calculated  now  to  provide  both  an  initial  a.s.sessmenl  of  mission  capability  and  also  provide  a 
historical  database  for  a  future  integrated  hardware/software  reliability  model. 

.Applicability  to  Hardware  Design  Reliability.  With  the  growing  use  of  hardware 
modeling  techniqties  such  as  VHDL  (Very  High-Speed  Integrated  Circuit  (VHSIC)  Hardware  De¬ 
scription  Language),  the  possibility  exists  that  software  reliability  measurement  (with  its  focus  on 
the  design  as  opposed  to  the  physical  aspect)  might  one  day  be  neces.sarily  applied  to  hardware 
designs  that  exist  only  in  the  memory  of  a  computer.  Toward  this  end,  soft  ware  reliability  models 


for  lOT&E  could  provide  the  foundation  for  deterniiiiing  tlie  lOTikE  logical  design  reliability  of 
hardware  from  components  to  systems. 

Integrated  Software  Reliahiltfy  Toots.  One  of  the  current  trends  in  software  reli¬ 
ability  is  to  have  many  different  reliability  models  integrated  into  one  tool.  Many  different  tools 
are  being  identified  to  perform  software  reliability  prediction,  measurement,  and  analysis,  and  it 
is  possible  that  not  all  software  reliability  models  are  applicable  to  all  pha.ses  of  the  software  life- 
cycle.  Indeed,  it  may  be  possible  or  even  <lesirable  to  implement  a  different  software  reliability 
model  during  each  phase  of  the  software  life-cycle  [69].  This  would  require  standardization  of  data 
to  be  used  between  models.  By  having  many  different  models  in  one  tool,  the  software  evaluator  in 
the  field  can  become  overburdened  with  uiiderstaiuliiig  the  intricacies  of  each  model  and  when  they 
apply,  as  well  as  possibly  collecting  data  that  couhl  vary  from  one  model  to  the  next.  An  example 
of  this  is  the  SMERFS  tool,  which  has  two  different  sets  of  models  selectable  from  the  main  menu, 
and  requires  different  types  of  data  for  each  set  [29].  Clearly,  having  one  standardized  model  (or  at 
least  .set)  with  one  basic  databa.se  will  make  software  reliability  evaluation  easier  for  the  software 
evaluator  in  the  field,  as  well  as  making  the  data  collection  job  easier  for  the  data  point  of  contact 
at  HQ  AFOTEC. 


Cost  Estimation.  This  thesis  proposes  using  the  candidate  models  to  determine 
the  time  needed  to  reach  a  desired  failure  inten.sity  objective  given  a  current  failure  intensity 
value.  A  recent  paper  lies  this  to  actual  testing  cost  [90].  The  paper  demonstrated  that,  due 
to  the  dependency  of  testing  costs  on  software  failure  behavior,  a  (|uantitative  cost  model  can  be 
incorporated  with  the  Logarithmic  Poisson  Execution  Time  model  to  determine  marginal  costs 
[90:423-424].  Additional  re.search  into  the  area  of  combining  software  cost  models  and  software 
reliability  models  could  then  provide  a  more  useful  tool  to  both  engineer  and  manager. 


6.3  Summary 

This  evaluation  reached  important  conclusions  about  the  application  of  software  reliability  to 
lOTAcE  of  weajion  systems.  It  is  clear  that  candidate  models  exist  which  can  work  with  sonie  degree 
of  certainly  in  evaluating  the  software  reliability,  and  hence,  the  operational  effectiveness  of  weapon 
system  software.  The  applicability  of  the.se  models  extends  far  beyond  the  IOT&:E  of  software,  and 
as  the  software  evaluation  process  matures  a  better  understanding  and  assessment  of  both  software 
and  the  overall  weapon  system  will  be  gained.  To  what  ever  extent  software  reliability  is  pursued, 
the  fact  that  it  is  being  considered  is  just  one  step  clo.ser  to  obtaining  “good  code”  for  the  user. 


Appendix  A.  Software  Definitions 


The  following  definitions  were  taken  from  multiple  sources,  and  are  included  here  as  additional 
information. 

Error.  Human  action  that  results  in  software  containing  a  fault.  Examples  include;  omission 
or  misinterpretation  of  user  requirements  in  a  software  specification,  and  incorrect  translation  or 
omission  of  a  requirement  in  the  design  specification. 

Fault.  A  manifestation  of  an  error  in  software.  A  fault,  if  encountered,  may  cause  a  failure. 
Synonym  -  Bug. 

Failure.  The  inability  of  a  system  or  system  component  to  perform  a  required  function 
within  specified  limits.  A  failure  may  be  produced  when  a  fault  is  encountered. 

Failure  Intensity.  The  number  of  failures  per  unit  time.  Failure  intensity  can  be  identified 
for  average  number  of  software  failures  per  flight  hour  (SF/FH)  and  average  number  of  software 
failures  per  mission  (SF/M). 

Maintainability.  The  ease  with  which  software  can  be  maintained.  The  extent  to  which  a 
component  facilitates  updating  to  satisfy  new  requirements  or  to  correct  deficiencies. 

Maturity.  The  extent  to  which  a  component  has  been  used  in  the  development  of  deliverable 
software  by  typical  users  and  to  which  feedback  from  that  use  has  been  reflected  in  modifications 
to  the  component. 

Moan  Time  to  Recover  Software.  The  amount  of  time  required  to  recover  from  a  software 
failure  and  restore  operational  capability  of  the  software.  This  could  be  the  time  necessary  to 
"reboot’’  the  system,  or  the  amount  of  time  spent  by  an  operator  clearing  an  error  display  and 
selecting  an  alternate  menu  option. 

McxUd.  A  representation  of  a  real  world  process,  device,  or  concept. 

Requirement.  A  condition  or  capability  that  must  be  met  or  possessed  by  a  system  or  system 
comi>onent  to  satisfy  a  contract,  standard,  specification  or  other  formally  imposed  document. 

Software  Maintenance.  Modification  of  a  software  product  after  delivery  to  correct  faults, 
to  improve  performance  or  other  attributes,  or  to  adapt  the  product  to  a  changed  environment. 

Software  Maturity.  An  assessment  of  the  software  based  on  the  number  of  faults  in  a 
computer  program.  This  includes  known  and  undiscovered  (latent)  faults.  Latent  faults  might 
not  be  discovered  until  several  years  after  full  scale  production,  if  at  all.  Emphasis  here  is  on 
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develoipmcni  acttviUes.  A  measure  of  the  software's  progress  in  its  evolution  toward  satisfaction  of 
all  documented  user  requirements. 

Software  Reliability.  The  probability  of  failure-hee  operation  of  a  computer  program  for 
a  specified  period  of  time.  The  emphasis  here  is  on  operational  activities.  If  the  software  fails,  then 
there  could  be  faults  that  must  be  corrected;  however,  not  all  faults  result  in  failures.  Software 
Reliability  Evaluation  can  be  divided  into  three  distinct  parts: 

•  Measurement.  Software  reliability  measurement  determines  the  present  failure  intensity, 
additional  failures  that  would  be  experienced  before  reaching  an  identified  failure  intensity 
objective,  and  additional  execution  time  necessary  to  reach  an  identified  failure  intensity 
objective. 

•  Prediction.  Software  reliability  prediction  attempts  to  determine  what  the  reliability  of 
software  will  be  at  some  time  t  from  a  present  software  reliability  measurement.  • 

•  Threshold.  The  level  of  software  reliability  identified  or  desired  by  the  decision  maker.  This 
can  be  expressed  as  a  reliability  number  (which  can  be  translated  with  respect  to  execution 
time)  or  a  failure  intensity  threshold  or  objective. 

Software  System  Effectiveness.  A  measure  of  the  percentage  of  time  the  software  system 
operates  correctly  (no  failures)  versus  the  total  attempted  operational  time.  The  SSE  can  be 
multiplied  by  the  Mission  Capable  (MC)  rate  (o  give  the  effect  of  software  on  Total  Weapon 
System  Effectiveness  (TWSE). 

Total  Weapon  System  Effectiveness.  The  Mission  Capable  (MC)  rate  for  a  weapon 
system  adjusted  to  account  for  the  effectiveness  of  the  software.  The  Software  System  Effectiveness 
(SSE)  can  be  mtiltiplied  by  the  MC  rate  to  determine  the  TWSE. 
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Appendix  B.  Software  Maturity  Data 


This  appendix  contains  the  reduced  data  set  from  the  initial  software  maturity  data  provided 
by  HQ  AFOTEC/LG5. 


B.I  Data  Sci  A1 


Database  lor  AIRCRFTl 

Date  #1  #2  #3  #4  #5  HSC  Total  Cum  Total 


10/01/86 

0 

0 

0 

0 

6 

0 

6 

6 

08/27/87 

1 

1 

0 

0 

2 

0 

4 

10 

08/28/87 

0 

2 

0 

0 

29 

0 

31 

41 

08/29/87 

0 

1 

0 

0 

37 

0 

38 

79 

08/31/87 

1 

0 

0 

0 

16 

0 

17 

96 

09/02/87 

0 

2 

0 

0 

25 

0 

27 

123 

09/03/87 

0 

0 

0 

0 

49 

0 

49 

172 

09/04/87 

0 

0 

0 

0 

58 

0 

58 

230 

09/05/87 

0 

2 

0 

1 

38 

0 

41 

271 

09/08/87 

0 

7 

0 

0 

69 

0 

76 

347 

09/09/87 

0 

0 

0 

0 

28 

0 

28 

375 

09/10/87 

0 

1 

0 

0 

24 

0 

25 

400 

09/11/87 

0 

0 

0 

0 

27 

0 

27 

427 

09/12/87 

0 

0 

0 

0 

32 

0 

32 

459 

09/14/87 

0 

10 

0 

0 

42 

0 

52 

511 

09/15/87 

0 

0 

0 

0 

16 

0 

16 

527 

09/16/87 

0 

2 

0 

0 

20 

0 

22 

549 

09/17/87 

0 

0 

0 

0 

27 

0 

27 

576 

09/18/87 

0 

0 

0 

0 

10 

0 

10 

586 

09/19/87 

0 

1 

0 

0 

50 

0 

51 

637 

09/21/87 

0 

2 

0 

0 

37 

0 

39 

676 

09/22/87 

0 

1 

0 

0 

35 

0 

36 

712 

09/23/87 

0 

0 

0 

0 

27 

0 

27 

739 

09/24/87 

0 

3 

0 

0 

29 

0 

32 

771 

09/25/87 

0 

1 

0 

0 

58 

0 

59 

830 

09/26/87 

0 

0 

0 

0 

41 

0 

41 

871 

09/28/87 

0 

1 

0 

0 

30 

0 

31 

902 

10/19/87 

0 

0 

0 

0 

14 

0 

14 

916 

10/20/87 

0 

0 

0 

0 

71 

0 

71 

987 

10/21/87 

0 

0 

0 

0 

30 

0 

30 

1017 

10/22/87 

0 

0 

0 

0 

18 

0 

18 

1035 

10/23/87 

0 

0 

0 

0 

81 

0 

81 

1116 

10/24/87 

0 

0 

0 

0 

9 

0 

9 

1125 

11/09/87 

0 

0 

0 

0 

13 

0 

13 

1138 

03/05/88 

0 

0 

1 

0 

0 

0 

1 

1139 

03/08/88 

0 

0 

1 

0 

21 

0 

22 

1161 

03/09/88 

0 

0 

1 

0 

21 

0 

22 

1183 

03/10/88 

0 

0 

3 

0 

8 

0 

11 

1194 

03/11/88 

0 

0 

1 

0 

5 

0 

6 

1200 

03/12/88 

0 

0 

2 

0 

31 

0 

33 

1233 

03/14/88 

0 

0 

4 

0 

17 

0 

21 

1254 

03/15/88 

0 

0 

0 

0 

5 

0 

5 

1259 

03/16/88 

0 

0 

1 

0 

22 

0 

23 

1282 

03/17/88 

0 

0 

2 

0 

13 

0 

15 

1297 

03/18/88 

0 

0 

0 

0 

13 

0 

13 

1310 

03/19/88 

0 

0 

0 

0 

9 

0 

9 

1319 

03/21/88 

0 

0 

1 

0 

14 

0 

15 

1334 

03/22/88 

0 

0 

2 

0 

13 

0 

15 

1349 

03/23/88 

0 

0 

0 

0 

5 

0 

5 

1354 

03/24/88 

0 

0 

0 

0 

6 

0 

6 

1360 

03/25/88 

0 

0 

0 

0 

6 

0 

6 

1366 

03/28/88 

0 

1 

1 

0 

36 

0 

38 

1404 

03/29/88 

0 

0 

2 

0 

28 

0 

30 

1434 

03/30/88 

0 

0 

2 

0 

34 

0 

36 

1470 

B  1 


04/26/88 

0 

0 

0 

0 

2 

0 

2 

1472 

04/27/88 

0 

0 

1 

0 

15 

0 

16 

1488 

04/28/88 

0 

0 

0 

0 

18 

0 

18 

1506 

04/29/88 

0 

0 

1 

0 

11 

0 

12 

1518 

04/30/88 

0 

0 

0 

0 

11 

0 

11 

1529 

B.2  Data  Set  A  2 


Date  #  1 


02/2A/87  0 
03/04/87  0 
05/20/87  0 
06/10/87  0 
06/25/87  0 
07/01/87  0 
08/03/87  0 
09/01/87  0 
09/03/87  0 
09/11/87  0 
09/17/87  0 
09/21/87  0 
11/04/87  0 
11/05/87  0 
11/12/87  0 
11/19/87  0 
11/20/87  0 
12/08/87  0 
12/09/87  0 
12/14/87  0 
01/04/88  0 
01/05/88  0 
01/14/88  0 
02/01/88  0 
02/23/88  0 
03/04/88  0 
03/10/88  0 
03/16/88  0 
03/17/88  0 
03/29/88  0 
04/12/88  0 
04/28/88  0 
05/03/88  0 
05/04/88  0 
05/06/88  0 
05/10/88  0 
05/12/88  0 
05/13/88  0 
05/16/88  0 
05/20/88  0 
05/23/88  0 
05/30/88  0 
06/03/88  0 
06/10/88  0 
06/13/88  0 
06/30/88  0 
07/01/88  0 
07/05/88  0 
07/11/88  0 
07/20/88  0 
09/16/88  0 
09/23/88  0 
09/27/88  0 
10/31/88  0 
11/14/88  0 
11/15/88  0 
12/01/88  0 
12/12/88  0 
12/23/88  0 
01/03/89  0 
01/05/89  0 
01/13/89  0 
01/18/89  0 
01/19/89  0 


Database  for  AIRCRFT2 
#2  #3  #4  #5  HSC  Total 


0  1 

1  0 

0  0 

0  3 

0  1 

0  1 

0  3 

0  1 

0  0 

1  1 

1  0 

0  3 

0  2 

0  0 

0  2 

0  1 

1  2 

0  0 

0  2 

0  1 

0  2 

0  2 

1  0 

0  1 

0  2 

0  1 

0  1 

0  0 

0  1 

0  1 

0  1 

0  1 

0  1 

0  2 

0  3 

0  1 

0  1 

0  1 

1  0 

0  1 

0  1 

0  0 

0  1 

0  2 

0  1 

0  1 

2  0 

0  0 

0  1 

0  1 

0  1 

0  0 

0  1 

0  1 

1  0 

0  1 

1  1 

1  0 

1  0 

0  2 

1  0 

0  0 

2  0 

1  0 


0  0 
0  0 
1  0 
0  0 
0  0 
0  0 
0  0 
0  0 
1  0 
0  0 
0  0 
0  0 
0  0 
1  0 
0  0 
0  0 
0  0 
1  0 
0  0 
1  0 
0  0 
0  0 
0  0 
0  0 
0  0 
0  0 
0  0 
1  0 
0  0 
0  0 
0  0 
0  0 
0  0 
1  0 
0  0 
0  0 
0  0 
0  0 
0  0 
0  0 
0  0 
1  0 
0  0 
0  0 
0  0 
1  0 
0  0 
1  0 
0  0 
0  0 
0  0 
1  0 
0  0 
0  0 
0  0 
0  0 
0  0 
0  0 
0  0 
0  0 
0  0 
1  0 
0  0 
0  0 


0  1 

0  1 

0  1 

0  3 

0  1 

0  1 

0  3 

0  1 

0  1 

0  2 

0  1 

0  3 

0  2 

0  1 

0  2 

0  1 

0  3 

0  1 

0  2 

0  2 

0  2 

0  2 

0  1 

0  1 

0  2 

0  1 

0  1 

0  1 

0  1 

0  1 

0  1 

0  1 

0  1 

0  3 

0  3 

0  1 

0  1 

0  1 

0  1 

0  1 

0  1 

0  1 

0  1 

0  2 

0  1 

0  2 

0  2 

0  1 

0  1 

0  1 

0  1 

0  1 

0  1 

0  1 

0  1 

0  1 

0  2 

0  1 

0  1 

0  2 

0  1 

0  1 

0  2 

0  1 


Cum  Total 


1 

2 

3 

6 

7 

8 
11 
12 
13 

15 

16 
19 
21 
22 

24 

25 
28 
29 
31 
33 
35 

37 

38 

39 

41 

42 

43 

44 

45 

46 

47 

48 

49 
52 

55 

56 

57 

58 

59 

60 
61 
62 
63 

65 

66 
68 

70 

71 

72 

73 

74 

75 

76 

77 

78 

79 
81 
82 
83 

85 

86 
87 

89 

90 


R-3 


01/20/89  0  0 
01/24/89  0  1 
01/26/89  0  0 
02/01/89  0  2 
02/02/89  0  0 
02/06/89  0  1 
02/15/89  0  0 
02/16/89  0  0 
02/18/89  0  1 
02/22/89  0  0 
02/24/89  0  0 
02/27/89  0  0 
03/04/89  0  0 
03/06/89  0  0 
03/13/89  0  2 
03/20/89  0  0 
03/27/89  0  0 
03/29/89  0  1 
04/19/89  0  0 
06/08/89  0  1 
06/13/89  0  0 
06/16/89  0  1 
06/21/89  0  0 
06/23/89  0  0 
07/07/89  0  1 
07/17/89  0  0 
07/21/89  0  0 
07/23/89  0  0 
07/25/89  0  0 


1  0  0  0  1 
1  0  0  0  2 
1  0  0  0  1 
0  0  0  0  2 
1  0  0  0  1 
1  0  0  0  2 
1  0  0  0  1 
1  0  0  0  1 
0  0  0  0  1 
1  0  0  0  1 
1  0  0  0  1 
2  0  0  0  2 
1  0  0  0  1 
1  0  0  0  1 
0  0  0  0  2 
1  0  0  0  1 
1  0  0  0  1 
0  0  0  0  1 
1  0  0  0  1 
0  0  0  0  1 
1  0  0  0  1 
1  0  0  0  2 
0  10  0  1 
110  0  2 
0  0  0  0  1 
1  0  0  0  1 
1  0  0  0  1 
1  0  0  0  1 
1  0  0  0  1 


91 

93 

94 

96 

97 
99 

100 

101 

102 

103 

104 
106 

107 

108 
110 
111 
112 

113 

114 

115 

116 
118 
119 
121 
122 

123 

124 

125 

126 


B.3  Data  Set  A3 


Database  for  AIRCRFT3 

Date  #1  #2  #3  #4  #5  NSC  Total 


1  0  0  0  0  1 
1110  0  3 


0  0  6 


12/29/87  0 
01/12/88  0 
02/25/88  3 
03/17/88  4 
03/22/88  0 
04/06/88  0 
04/11/88  1 
04/20/88  2 
04/21/88  2 
04/26/88  0 
04/28/88  1 
05/25/88  0 
06/01/88  0 
06/02/88  0 
06/13/88  1 
07/01/88  1 
07/14/88  1 
07/20/88  1 
07/21/88  0 
07/28/88  1 
08/04/88  0 
08/05/88  2 
08/10/88  3 
08/12/88  0 
08/15/88  1 
08/17/88  2 
08/18/88  0 
08/24/88  1 
08/25/88  0 
08/26/88  2 
08/30/88  2 
08/31/88  1 
09/01/88  4 
09/02/88  2 
09/06/88  0 
09/09/88  1 
09/12/88  0 
09/13/88  4 
09/14/88  0 
09/15/88  1 
09/16/88  2 
09/19/88  1 
09/20/88  0 
09/21/88  0 
09/27/88  2 
09/29/88  0 
10/04/88  0 
10/07/88  2 
11/02/88  0 
11/03/88  1 
11/07/88  0 
11/15/88  1 
11/21/88  0 
11/22/88  1 
11/28/88  2 
11/29/88  1 
12/02/88  1 
12/05/88  0 
12/06/88  0 
12/12/88  4 
12/20/88  1 
12/22/88  0 
12/27/88  1 
12/28/88  6 


0 
0 

1  0  0 

0  0  1 

0  0  0 

1  0  0 

0  0  0 

0  1  0 

2  0  0 

10  0 

3  0  0 

1  0  0 

1  0  0 

3  0  0 

0  0  0 

0  0  0 

3  0  0 

2  0  0 

10  0 

5  1  0 

3  0  0 

1  0  0 

1  1  0 

2  0  0 

0  10 

1  0  0 

2  0  0 

2  0  0 

7  0  0 

1  0  0 

0  0  0 

3  0  0 

2  0  0 

3  0  0 

1  0  0 

0  0  0 

3  0  0 

4  2  0 

2  0  0 

0  0  0 

1  0  0 

2  0  0 

0  0  0 

1  1  0 

2  0  0 

1  0  0 

2  0  0 

1  0  0 

0  1  0 

0  0  0 

1  0  0 

1  0  0 

0  0  0 

3  0  0 

1  0  0 

2  10 

2  0  0 

0  0  0 

3  0  0 

1  1  0 

0  0  0 

10  1  0 


0  0  8 

0  0  1 

0  0  1 

0  0  1 

0  0  3 

0  0  2 

0  0  1 

0  0  3 

0  0  1 

0  0  3 

0  0  1 

0  0  2 

0  0  4 

0  0  1 

0  0  1 

0  0  3 

0  0  3 

0  0  1 

0  0  8 

0  0  6 

0  0  1 

0  0  3 

0  0  4 

0  0  1 

0  0  2 

0  0  2 

0  0  4 

0  0  9 

0  0  2 

0  0  4 

0  0  5 

0  0  2 

0  0  4 

0  0  1 

0  0  4 

0  0  3 

0  0  7 

0  0  4 

0  0  1 

0  0  1 

0  0  2 

0  0  2 

0  0  2 

0  0  2 

0  0  3 

0  0  2 

0  0  2 

0  0  1 

0  0  1 

0  0  1 

0  0  2 

0  0  2 

0  0  4 

0  0  2 

0  0  3 

0  0  2 

0  0  4 

0  0  4 

0  0  2 

0  0  1 

0  0  17 


2 

3 


Cum  Total 


1 

4 

10 

18 

19 

20 
21 
24 
26 
27 

30 

31 

34 

35 
37 

41 

42 

43 
46 

49 

50 
58 

64 

65 
68 

72 

73 
75 
77 
81 
90 
92 
96 

101 

103 

107 

108 
112 
115 
122 
126 

127 

128 
130 
132 
134 
136 
139 
141 

143 

144 

145 

146 
148 
150 
154 
156 
159 
161 
165 
169 

171 

172 
189 


B-r, 


01/03/89  0 
01/04/89  0 
01/09/89  1 
01/13/89  1 
01/23/89  0 
01/30/89  2 
02/14/89  0 
02/17/89  3 
02/27/89  0 
02/28/89  0 
03/01/89  2 
03/09/89  0 
03/10/89  0 
03/14/89  0 
03/15/89  0 
03/16/89  3 
03/22/89  6 
03/27/89  0 
04/20/89  0 
04/26/89  0 
04/27/89  0 
05/04/89  0 
05/09/89  2 
05/10/89  1 
05/15/89  0 
05/16/89  1 
05/18/89  0 
05/23/89  0 
05/25/89  0 
06/01/89  0 
06/05/89  0 
06/06/89  0 
06/07/89  2 
06/15/89  1 
06/16/89  0 
06/21/89  1 
06/22/89  1 
06/23/89  0 
06/26/89  1 
06/27/89  0 
06/29/89  0 
07/05/89  1 
07/10/89  0 
07/13/89  0 
08/01/89  3 
08/03/89  0 
08/10/89  1 
08/15/89  1 
08/22/89  0 
08/23/89  0 
08/29/89  1 
08/31/89  1 
09/11/89  0 
09/15/89  0 
09/22/89  0 


0  10 

1  0  0 

0  0  0 

0  0  0 

1  0  0 

0  0  0 

1  0  0 

0  0  0 

1  0  0 

1  0  0 

0  0  0 

1  0  0 

2  0  0 

2  1  0 

2  0  0 

0  0  0 

4  0  0 

1  0  0 

1  0  0 

0  1  0 

10  0 

1  0  0 

2  0  0 

0  0  0 

1  0  0 

0  0  0 

1  0  0 

1  0  0 

10  0 

1  1  0 

2  0  0 

1  0  0 

1  0  0 

1  0  0 

1  0  0 

0  0  0 

1  0  0 

3  1  0 

0  2  0 

1  0  0 

1  0  0 

1  0  0 

10  0 

10  0 

1  3  0 

1  0  0 

1  0  0 

0  0  0 

3  0  0 

0  1  0 

1  0  0 

0  0  0 

1  0  0 

1  0  0 

1  0  0 


0  0  1 

0  0  1 

0  0  1 

0  0  1 

0  0  1 

0  0  2 

0  0  1 

0  0  3 

0  0  1 

0  0  1 

0  0  2 

0  0  1 

0  0  2 

0  0  3 

0  0  2 

0  0  3 

0  0  10 

0  0  1 

0  0  1 

0  0  1 

0  0  1 

0  0  1 

0  0  4 

0  0  1 

0  0  1 

0  0  1 

0  0  1 

0  0  1 

0  0  1 

0  0  2 

0  0  2 

0  0  1 

0  0  3 

0  0  2 

0  0  1 

0  0  1 

0  0  2 

0  0  4 

0  0  3 

0  0  1 

0  0  1 

0  0  2 

0  0  1 

0  0  1 

0  0  7 

0  0  1 

0  0  2 

0  0  1 

0  0  3 

0  0  1 

0  0  2 

0  0  1 

0  0  1 

0  0  1 

0  0  1 


190 

191 

192 

193 

194 

196 

197 
200 
201 
202 

204 

205 
207 
210 
212 
215 

225 

226 

227 

228 

229 

230 

234 

235 

236 

237 

238 

239 

240 
242 

244 

245 
248 

250 

251 

252 
254 
258 
261 
262 
263 

265 

266 
267 

274 

275 

277 

278 
281 
282 

284 

285 

286 

287 

288 


B-6 


B.4  Data  Set  Si 


Database  for  SPACEl 

Date  #1  #2  #3  #4  #5  KSC  Total  Cum  Total 


01/10/86  01000 
01/15/86  01000 

01/30/86  01000 

02/10/86  01000 
02/20/86  01000 
03/03/86  01000 

03/05/86  04000 

03/11/86  01000 

03/24/86  01000 

03/26/86  01000 

03/28/86  01000 

03/31/86  01000 

04/02/86  01000 

04/07/86  01000 

04/08/86  02000 

04/09/86  01000 

04/10/86  01000 

04/11/86  01000 

04/12/86  02000 

04/14/86  01000 

04/17/86  01000 

04/22/86  01000 

04/28/86  02000 

04/30/86  01000 

05/06/86  01000 

05/07/86  01000 

05/08/86  01000 

05/12/86  03000 

05/13/86  02000 

05/18/86  04000 

05/19/86  01000 

05/20/86  02000 

05/21/86  01000 

05/28/86  01000 

05/29/86  01000 

05/30/86  01000 

06/02/86  03000 

06/04/86  01000 

06/05/86  01000 

06/06/86  01000 
06/11/86  02000 
06/13/86  01000 

06/14/86  09000 

06/18/86  01000 
06/24/86  05000 

06/25/86  02000 

06/29/86  02000 

07/03/86  01000 

07/07/86  04000 

07/08/86  0  1  0  1  0 

07/09/86  01000 

07/10/86  01000 

07/11/86  05000 

07/14/86  01000 

07/15/86  02000 

07/16/86  07000 

07/18/86  01000 

07/22/86  02000 

07/23/86  01000 

07/29/86  03000 

08/01/86  02000 
08/02/86  01000 
08/04/86  0  11  0  0  0 

08/05/86  03000 


0  1 

0  1 

0  1 

0  1 

0  1 

0  1 

0  4 

0  1 

0  1 

0  1 

0  1 

0  1 

0  1 

0  1 

0  2 

0  1 

0  1 

0  1 

0  2 

0  1 

0  1 

0  1 

0  2 

0  1 

0  1 

0  1 

0  1 

0  3 

0  2 

0  4 

0  1 

0  2 

0  1 

0  1 

0  1 

0  1 

0  3 

0  1 

0  1 

0  1 

0  2 

0  1 

0  9 

0  1 

0  5 

0  2 

0  2 

0  1 

0  4 

0  1 

0  1 

0  1 

0  5 

0  1 

0  2 

0  7 

0  1 

0  2 

0  1 

0  3 

0  2 

0  1 

0  11 

0  3 


1 

2 

3 

4 

5 

6 
10 
11 
12 

13 

14 

15 

16 
17 

19 

20 
21 
22 

24 

25 

26 
27 

29 

30 

31 

32 

33 
36 
38 

42 

43 

45 

46 

47 

48 

49 

52 

53 

54 

55 

57 

58 

67 

68 
73 
75 

77 

78 
82 

83 

84 

85 

90 

91 
93 

100 

101 

103 

104 
107 
109 

no 

121 

124 


B-7 


08/06/86  0  2 

08/07/86  0  2 

08/09/86  0  1 

08/12/86  0  1 

08/13/86  0  2 

08/14/86  0  1 

08/15/86  0  2 

08/19/86  0  3 

08/20/86  0  2 

08/21/86  0  8 

08/22/86  0  6 

08/25/86  0  1 

08/26/86  0  1 

09/02/86  0  2 

09/03/86  0  3 

09/04/86  0  3 

09/05/86  0  1 

09/08/86  0  2 

09/09/86  0  15 

09/10/86  0  1 

09/11/86  0  1 

09/12/86  0  1 

09/15/86  0  3 

09/16/86  0  1 

09/17/86  0  6 

09/18/86  0  2 

09/19/86  0  4 

09/22/86  0  6 

09/23/86  0  1 

09/25/86  0  1 

09/26/86  0  2 

09/29/86  0  1 

09/30/86  0  5 

10/01/86  0  1 

10/02/86  0  2 

10/03/86  0  2 

10/05/86  0  1 

10/06/86  0  4 

10/07/86  0  2 

10/09/86  0  1 

10/13/86  0  5 

10/14/86  0  6 

10/15/86  0  9 

10/16/86  0  1 

10/20/86  0  1 

10/21/86  0  3 

10/22/86  0  1 

10/23/86  0  6 

10/24/86  0  1 

10/27/86  0  1 

10/28/86  0  6 

10/29/86  0  2 

10/30/86  0  1 

11/03/86  0  10 

11/04/86  0  9 

11/05/86  0  3 

11/06/86  0  2 

11/07/86  0  3 

11/08/86  0  1 

11/10/86  0  22 

11/11/86  0  3 

11/12/86  0  3 

11/13/86  0  3 

11/14/86  0  3 

11/17/86  0  1 

11/18/86  0  2 
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1763 

10/01/87 

0 

8 

0 

0 

0 

0 

8 

1771 

10/02/87 

0 

2 

0 

0 

0 

0 

2 

1773 

10/03/87 

0 

1 

0 

0 

0 

0 

1 

1774 

10/04/87 

0 

2 

0 

0 

0 

0 

2 

1776 

10/05/87 

0 

11 

0 

0 

0 

0 

11 

1787 

10/06/87 

0 

8 

0 

0 

0 

0 

8 

1795 

10/07/87 

0 

5 

0 

0 

0 

0 

5 

1800 

10/08/87 

0 

6 

0 

0 

0 

0 

6 

1806 

10/09/87 

0 

3 

0 

0 

0 

0 

3 

1809 

10/12/87 

0 

6 

0 

0 

0 

0 

6 

1815 

10/13/87 

0 

5 

0 

0 

0 

0 

5 

1820 

10/14/87 

0 

3 

0 

0 

0 

0 

3 

1823 

10/15/87 

0 

12 

0 

0 

0 

0 

12 

1835 

10/16/87 

0 

1 

0 

0 

0 

0 

1 

1836 

10/19/87 

0 

3 

0 

0 

0 

0 

3 

1839 

10/20/87 

0 

6 

0 

0 

0 

0 

6 

1845 

10/21/87 

0 

5 

0 

0 

0 

0 

5 

1850 

10/22/87 

0 

5 

0 

0 

0 

0 

5 

1855 

10/23/87 

0 

6 

0 

0 

0 

0 

6 

1861 

10/26/87 

0 

14 

0 

0 

0 

0 

14 

1875 

10/27/87 

0 

2 

0 

0 

0 

0 

2 

1877 

10/28/87 

0 

3 

0 

0 

0 

0 

3 

1880 

10/29/87 

0 

2 

0 

0 

0 

0 

2 

1882 

10/30/87 

0 

3 

0 

0 

0 

0 

3 

1885 

11/02/87 

0 

2 

0 

0 

0 

0 

2 

1887 

11/03/87 

0 

7 

0 

0 

0 

0 

7 

1894 

11/04/87 

0 

5 

0 

0 

0 

0 

5 

1899 

11/05/87 

0 

3 

0 

0 

0 

0 

3 

1902 

11/06/87 

0 

6 

0 

0 

0 

0 

6 

1908 

11/09/87 

0 

6 

0 

0 

0 

0 

6 

1914 

11/10/87 

0 

5 

0 

0 

0 

0 

5 

1919 

11/11/87 

0 

6 

0 

0 

0 

0 

6 

1925 

11/12/87 

0 

2 

0 

0 

0 

0 

2 

1927 

11/13/87 

0 

1 

0 

0 

0 

0 

1 

1928 

11/16/87 

0 

6 

0 

0 

0 

0 

6 

1934 

11/17/87 

0 

6 

0 

0 

0 

0 

6 

1940 

11/18/87 

0 

1 

0 

0 

0 

0 

1 

1941 

11/19/87 

0 

14 

0 

0 

0 

0 

14 

1955 

11/20/87 

0 

4 

0 

0 

0 

0 

4 

1959 

11/23/87 

0 

7 

0 

0 

0 

0 

7 

1966 

11/24/87 

0 

1 

0 

0 

0 

0 

1 

1967 

11/25/87 

0 

4 

0 

0 

0 

0 

4 

1971 

11/30/87 

0 

7 

0 

0 

0 

0 

7 

1978 

12/01/87 

0 

7 

0 

0 

0 

0 

7 

1985 

12/02/87 

0 

1 

0 

0 

0 

0 

1 

1986 

12/03/87 

0 

5 

0 

0 

0 

0 

5 

1991 

12/04/87 

0 

1 

0 

0 

0 

0 

1 

1992 

12/07/87 

0 

6 

0 

0 

0 

0 

6 

1998 

12/08/87 

0 

1 

0 

0 

0 

0 

1 

1999 

12/09/87 

0 

3 

0 

0 

0 

0 

3 

2002 

12/10/87 

0 

6 

0 

0 

0 

0 

6 

2008 

12/11/87 

0 

3 

0 

0 

0 

0 

3 

2011 

12/12/87 

0 

2 

0 

0 

0 

0 

2 

2013 

12/14/87 

0 

5 

0 

0 

0 

0 

5 

2018 

12/16/87 

0 

2 

0 

0 

0 

0 

2 

2020 

12/17/87 

0 

1 

0 

0 

0 

0 

1 

2021 

12/18/87 

0 

1 

0 

0 

0 

0 

1 

2022 

12/19/87 

0 

1 

0 

0 

0 

0 

1 

2023 

12/21/87 

0 

1 

0 

0 

0 

0 

1 

2024 

12/22/87 

0 

5 

0 

0 

0 

0 

5 

2029 

12/23/87 

0 

2 

0 

0 

0 

0 

2 

2031 

n  1-2 


01/01/88 

0 

2 

0 

0 

0 

0 

2 

2033 

01/04/88 

0 

2 

0 

0 

0 

0 

2 

2035 

01/05/88 

0 

1 

0 

0 

0 

0 

1 

2036 

01/06/88 

0 

7 

0 

0 

0 

0 

7 

2043 

01/07/88 

0 

4 

0 

0 

0 

0 

4 

2047 

01/08/88 

0 

1 

0 

0 

0 

0 

1 

2048 

01/09/88 

0 

4 

0 

0 

0 

0 

4 

2052 

01/11/88 

0 

2 

0 

0 

0 

0 

2 

2054 

01/12/88 

0 

13 

0 

0 

0 

0 

13 

2067 

01/13/88 

0 

6 

0 

0 

0 

0 

6 

2073 

01/14/88 

0 

1 

0 

0 

0 

0 

1 

2074 

01/15/88 

0 

2 

0 

0 

0 

0 

2 

2076 

01/18/88 

0 

1 

0 

0 

0 

0 

1 

2077 

01/19/88 

0 

4 

0 

0 

0 

0 

4 

2081 

01/20/88 

0 

2 

0 

0 

0 

0 

2 

2083 

01/22/88 

0 

3 

0 

0 

0 

0 

3 

2086 

01/23/88 

0 

2 

0 

0 

0 

0 

2 

2088 

01/25/88 

0 

2 

0 

0 

0 

0 

2 

2090 

01/26/88 

0 

7 

0 

0 

0 

0 

7 

2097 

01/27/88 

0 

2 

0 

0 

0 

0 

2 

2099 

01/28/88 

0 

3 

0 

0 

0 

0 

3 

2102 

01/29/88 

0 

9 

0 

0 

0 

0 

9 

2111 

01/31/88 

0 

1 

0 

0 

0 

0 

1 

2112 

02/03/88 

0 

5 

0 

0 

0 

0 

5 

2117 

02/05/88 

0 

10 

0 

0 

0 

0 

10 

2127 

02/09/88 

0 

15 

0 

0 

0 

0 

15 

2142 

02/10/88 

0 

1 

0 

0 

0 

0 

1 

2143 

02/11/88 

0 

10 

0 

0 

0 

0 

10 

2153 

02/12/88 

0 

20 

0 

0 

0 

0 

20 

2173 

02/15/88 

0 

2 

0 

0 

0 

0 

2 

2175 

02/16/88 

0 

2 

0 

0 

0 

0 

2 

2177 

02/17/88 

0 

5 

0 

0 

0 

0 

5 

2182 

02/18/88 

0 

3 

0 

0 

0 

0 

3 

2185 

02/19/88 

0 

3 

0 

0 

0 

0 

3 

2188 

02/21/88 

0 

1 

0 

0 

0 

0 

1 

2189 

02/22/88 

0 

2 

0 

0 

0 

0 

2 

2191 

02/23/88 

0 

2 

0 

0 

0 

0 

2 

2193 

02/24/88 

0 

4 

0 

0 

0 

0 

4 

2197 

02/25/88 

0 

2 

0 

0 

0 

0 

2 

2199 

02/26/88 

0 

3 

0 

0 

0 

0 

3 

2202 

02/29/88 

0 

4 

0 

0 

0 

0 

4 

2206 

03/01/88 

0 

2 

0 

0 

0 

0 

2 

2208 

03/02/88 

0 

3 

0 

0 

0 

0 

3 

2211 

03/03/88 

0 

8 

0 

0 

0 

0 

8 

2219 

03/04/88 

0 

1 

0 

0 

0 

0 

1 

2220 

03/05/88 

0 

1 

0 

0 

0 

0 

1 

2221 

03/07/88 

0 

1 

0 

0 

0 

0 

1 

2222 

03/08/88 

0 

2 

0 

0 

0 

0 

2 

2224 

03/09/88 

0 

5 

0 

0 

0 

0 

5 

2229 

03/10/88 

0 

3 

0 

0 

0 

0 

3 

2232 

03/11/88 

0 

3 

0 

0 

0 

0 

3 

2235 

03/12/88 

0 

2 

0 

0 

0 

0 

2 

2237 

03/15/88 

0 

3 

0 

0 

0 

0 

3 

2240 

03/16/88 

0 

9 

0 

0 

0 

0 

9 

2249 

03/17/88 

0 

13 

0 

0 

0 

0 

13 

2262 

03/18/88 

0 

3 

0 

0 

0 

0 

3 

2265 

03/19/88 

0 

1 

0 

0 

0 

0 

1 

2266 

03/20/88 

0 

3 

0 

0 

0 

0 

3 

2269 

03/21/88 

0 

2 

0 

0 

0 

0 

2 

2271 

03/22/88 

0 

1 

0 

0 

0 

0 

1 

2272 

03/23/88 

0 

3 

0 

0 

0 

0 

3 

2275 

03/24/88 

0 

3 

0 

0 

0 

0 

3 

2278 

03/25/88 

0 

2 

0 

0 

0 

0 

2 

2280 

03/28/88 

0 

1 

0 

0 

0 

0 

1 

2281 

03/30/88 

0 

4 

0 

0 

0 

0 

4 

2285 

03/31/88 

0 

2 

0 

0 

0 

0 

2 

2287 

04/01/88 

0 

7 

0 

0 

0 

0 

7 

2294 

04/05/88 

0 

6 

0 

0 

0 

0 

6 

2300 

04/06/88 

0 

4 

0 

0 

0 

0 

4 

2304 

04/07/88 

0 

7 

0 

0 

0 

0 

7 

2311 

04/08/88  0 
04/11/88  0 
04/12/88  0 
04/13/88  0 
04/14/88  0 
04/15/88  0 
04/16/88  0 
04/17/88  0 
04/19/88  0 
04/21/88  0 
04/22/88  0 
04/25/88  0 
04/26/88  0 
04/27/88  0 
04/28/88  0 
04/29/88  0 
05/01/88  0 
05/02/88  0 
05/03/88  0 
05/05/88  0 
05/06/88  0 
05/07/88  0 
05/09/88  0 
05/11/88  0 
05/13/88  0 
05/15/88  0 
05/19/88  0 
05/20/88  0 
05/25/88  0 
05/26/88  0 
05/27/88  0 
06/01/88  0 
06/02/88  0 
06/06/88  0 
06/09/88  0 
06/10/88  0 
06/14/88  0 
06/15/88  0 
06/16/88  0 
06/17/88  0 
06/20/88  0 
06/21/88  0 
06/22/88  0 
06/24/88  0 
06/27/88  0 
06/28/88  0 
06/29/88  0 
07/01/88  0 
07/04/88  0 
07/05/88  0 
07/06/88  0 
07/07/88  0 
07/08/88  0 
07/11/88  0 
07/12/88  0 
07/14/88  0 
07/15/88  0 
07/18/88  0 
07/20/88  0 
07/21/88  0 
07/22/88  0 
07/25/88  0 
07/26/88  0 
07/27/88  0 
07/29/88  0 
08/03/88  0 
08/05/88  0 
08/09/88  0 
08/12/88  0 
08/15/88  0 


9  0  0 

2  0  0 

4  0  0 

10  0 

2  0  0 

1  0  0 

1  0  0 

10  0 

4  0  0 

3  0  0 

2  0  0 

6  0  0 

2  0  0 

3  0  0 

2  0  0 

2  0  0 

5  0  0 

2  0  0 

2  0  0 

3  0  0 

10  0 

2  0  0 

3  0  0 

3  0  0 

10  0 

1  0  0 

4  0  0 

10  0 

2  0  0 

2  0  0 

2  0  0 

3  0  0 

2  0  0 

4  0  0 

1  0  0 

1  0  0 

2  0  0 

1  0  0 

2  0  0 

1  0  0 

2  0  0 

1  0  0 

2  0  0 

1  0  0 

4  0  0 

7  0  0 

3  0  0 

1  0  0 

2  0  0 

1  0  0 

1  0  0 

2  0  0 

1  0  0 

3  0  0 

3  0  0 

4  0  0 

4  0  0 

2  0  0 

2  0  0 

2  0  0 

10  0 

3  0  0 

1  0  0 

2  0  0 

2  0  0 

3  0  0 

1  0  0 

1  0  0 

1  0  0 

1  0  0 


0  0  9 
0  0  2 
0  0  4 
0  0  1 
0  0  2 
0  0  1 
0  0  1 
0  0  1 
0  0  4 
0  0  3 
0  0  2 
0  0  6 
0  0  2 
0  0  3 
0  0  2 
0  0  2 
0  0  5 
0  0  2 
0  0  2 
0  0  3 
0  0  1 
0  0  2 
0  0  3 
0  0  3 
0  0  1 
0  0  1 
0  0  4 
0  0  1 
0  0  2 
0  0  2 
0  0  2 
0  0  3 
0  0  2 
0  0  4 
0  0  1 
0  0  1 
0  0  2 
0  0  1 
0  0  2 
0  0  1 
0  0  2 
0  0  1 
0  0  2 
0  0  1 
0  0  4 
0  0  7 
0  0  3 
0  0  1 
0  0  2 
0  0  1 
0  0  1 
0  0  2 
0  0  1 
0  0  3 
0  0  3 
0  0  4 
0  0  4 
0  0  2 
0  0  2 
0  0  2 
0  0  1 
0  0  3 
0  0  1 
0  0  2 
0  0  2 
0  0  3 
0  0  1 
0  0  1 
0  0  1 
0  0  1 


2320 

2322 

2326 

2327 

2329 

2330 

2331 

2332 
2336 
2339 
2341 
2347 
2349 
2352 
2354 
2356 
2361 
2363 
2365 

2368 

2369 
2371 
2374 

2377 

2378 

2379 

2383 

2384 
2386 
2388 
2390 
2393 
2395 

2399 

2400 

2401 

2403 

2404 

2406 

2407 

2409 

2410 

2412 

2413 
2417 
2424 

2427 

2428 

2430 

2431 

2432 

2434 

2435 
2438 
2441 
2445 
2449 
2451 
2453 

2455 

2456 
2459 
246u 
2462 
2464 

2467 

2468 

2469 

2470 

2471 


B-M 


08/17/88 

0 

1 

0 

0 

0 

0 

1 

2472 

08/18/88 

0 

2 

0 

0 

0 

0 

2 

2474 

08/19/88 

0 

2 

0 

0 

0 

0 

2 

2476 

08/24/88 

0 

2 

0 

0 

0 

0 

2 

2478 

08/25/88 

0 

2 

0 

0 

0 

0 

2 

2480 

08/26/88 

0 

1 

0 

0 

0 

0 

1 

2481 

08/29/88 

0 

1 

0 

0 

0 

0 

1 

2482 

08/30/88 

0 

4 

0 

0 

0 

0 

4 

2486 

09/01/88 

0 

1 

0 

0 

0 

0 

1 

2487 

09/02/88 

0 

2 

0 

0 

0 

0 

2 

2489 

09/08/88 

0 

2 

0 

0 

0 

0 

2 

2491 

09/11/88 

0 

1 

0 

0 

0 

0 

1 

2492 

09/12/88 

0 

1 

0 

0 

0 

0 

1 

2493 

09/13/88 

0 

1 

0 

0 

0 

0 

1 

2494 

09/14/88 

0 

1 

0 

0 

0 

0 

1 

2495 

09/15/88 

0 

1 

0 

0 

0 

0 

1 

2496 

09/16/88 

0 

1 

0 

0 

0 

0 

1 

2497 

09/19/88 

0 

1 

0 

0 

0 

0 

1 

2498 

09/20/88 

0 

1 

0 

0 

0 

0 

1 

2499 

09/21/88 

0 

1 

0 

0 

0 

0 

1 

2500 

09/23/88 

0 

2 

0 

0 

0 

0 

2 

2502 

09/28/88 

0 

1 

0 

0 

0 

0 

1 

2503 

10/05/88 

0 

3 

0 

0 

0 

0 

3 

2506 

10/11/88 

0 

1 

0 

0 

0 

0 

1 

2507 

10/14/88 

0 

1 

0 

0 

0 

0 

1 

2508 

10/18/88 

0 

1 

0 

0 

0 

0 

1 

2509 

11/08/88 

0 

1 

0 

0 

0 

0 

1 

2510 

11/16/88 

0 

2 

0 

0 

0 

0 

2 

2512 

11/17/88 

0 

1 

0 

0 

0 

0 

1 

2513 

11/22/88 

0 

1 

0 

0 

0 

0 

1 

2514 

12/12/88 

0 

1 

0 

0 

0 

0 

1 

2515 

12/13/88 

0 

1 

0 

0 

0 

0 

1 

2516 

01/30/89 

0 

1 

0 

0 

0 

0 

1 

2517 

02/06/89 

0 

1 

0 

0 

0 

0 

1 

2518 

02/13/89 

0 

3 

0 

0 

0 

0 

3 

2521 

04/01/89 

0 

1 

0 

0 

0 

0 

1 

2522 

04/18/89 

0 

1 

0 

0 

0 

0 

1 

2523 

05/10/89 

0 

2 

0 

0 

0 

0 

2 

2525 

B-15 


B.5  Data  Sti  \Vl 


Database  for  WSTl 


Date  # 

1  « 

2  < 

3  « 

4 

02/12/90 

0 

0 

0 

2 

03/26/90 

0 

0 

35 

0 

03/29/90 

0 

0 

0 

1 

03/30/90 

0 

0 

0 

2 

04/02/90 

0 

0 

0 

3 

04/03/90 

0 

0 

0 

7 

04/05/90 

0 

0 

0 

5 

04/10/90 

0 

0 

0 

7 

04/11/90 

0 

0 

0 

1 

04/12/90 

0 

0 

0 

1 

04/18/90 

0 

0 

0 

15 

04/19/90 

0 

0 

0 

10 

04/20/90 

0 

0 

2 

10 

04/23/90 

0 

0 

3 

3 

04/24/90 

0 

0 

0 

6 

04/25/90 

0 

0 

0 

14 

04/26/90 

0 

0 

0 

3 

04/27/90 

0 

0 

0 

3 

04/30/90 

0 

0 

0 

7 

05/01/90 

0 

0 

0 

1 

05/02/90 

0 

0 

0 

9 

05/03/90 

0 

0 

1 

20 

05/04/90 

0 

0 

0 

3 

05/08/90 

0 

0 

0 

8 

05/09/90 

0 

0 

2 

9 

05/22/90 

0 

0 

1 

7 

05/23/90 

0 

0 

0 

7 

05/29/90 

0 

0 

1 

0 

05/30/90 

0 

0 

0 

21 

05/31/90 

0 

0 

0 

5 

06/01/90 

0 

0 

0 

5 

06/04/90 

0 

0 

0 

9 

06/05/90 

0 

0 

0 

6 

06/06/90 

0 

0 

0 

12 

06/07/90 

0 

0 

0 

12 

06/14/90 

0 

1 

3 

15 

06/15/90 

0 

0 

0 

22 

06/18/90 

0 

0 

0 

19 

06/19/90 

0 

0 

1 

2 

06/21/90 

0 

0 

3 

31 

06/22/90 

0 

0 

1 

0 

06/27/90 

0 

0 

0 

6 

06/28/90 

0 

0 

3 

7 

06/29/90 

0 

0 

2 

28 

07/31/90 

0 

0 

1 

0 

08/01/90 

0 

0 

2 

0 

5  NSC  Total  Total 

0  0  2  2 


0 

0 

35 

37 

2 

0 

3 

40 

4 

0 

6 

46 

0 

0 

3 

49 
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0 

7 

56 

0 

0 

5 

61 

0 

0 

7 

68 

0 

0 

1 

69 

0 

0 

1 

70 

0 

0 

15 

85 

0 

0 

10 

95 

0 

0 

12 

107 

1 

0 

7 

114 

0 

0 

6 

120 

0 

0 

14 

134 

0 

0 

3 

137 

0 

0 

3 

140 

0 

0 

7 

147 

0 

0 

1 

148 

1 

0 

10 

158 

0 

0 

21 

179 

0 

0 

3 

182 

0 

0 

8 

190 

0 

0 

11 

201 

0 

0 

8 

209 

0 

0 

7 

216 

0 

0 

1 

217 

0 

0 

21 

238 

0 

0 

5 

243 

0 

0 

5 

248 

0 

0 

9 

257 

0 

0 

6 

263 

0 

0 

12 

275 

0 

0 

12 

287 

1 

0 

20 

307 

3 

0 

25 

332 

2 

0 

21 

353 

0 

0 

3 

356 

4 

0 

38 

394 

0 

0 

1 

395 

0 

0 

6 

401 

0 

0 

10 

411 

6 

0 

36 

447 

0 

0 

1 

448 

0 

0 

2 

450 
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Appendix  C.  Dttaihd  Analysis  and  Design 


This  appendix  contains  tlie  detailed  analysis  and  design  of  software  to  implement  the  candi¬ 
date  software  reliability  models. 


C.l  Background 

Five  categories  of  software  failures  exist,  ranging  from  critical  to  noncritical,  each  one  de¬ 
scribed  in  terms  of  mission  success  [86:8-2].  These  categories  have  been  applied  to  lOTAcE.  with 
the  following  software  failure  severity  levels  applied  (23:14): 


•  System  Abort.  Severity  Level  1.  Software  or  firmware  problem  that  results  in  a 
system  abort. 

•  System  Degraded  No  Workaround.  Severity  Level  2.  Software  or  firmware 
problem  that  degrades  the  system  and  no  alternative  workaround  exists  (program 
restarts  not  acceptable). 

•  System  Degraded  Workaround.  Severity  Level  3.  Software  or  firmware  prob¬ 
lem  that  degrades  the  system  and  there  exists  an  alternative  workaround  (e.g., 
system  rerouting  through  operator  switchology;  program  restart  not  acceptable). 

•  System  Not  Degraded.  Severity  Level  4.  An  indicated  software  or  firmware 
problem  that  does  not  degrade  the  system  or  any  essential  system  function. 

•  Minor  Fault.  Severity  Level  5.  .Ml  other  minor  nonfunctional  software  deficien¬ 
cies. 


Currently,  most  software  reliability  models  assume  either  all  errors  have  the  same  weight  (or 
severity  level)  or  the  weighting  is  based  on  observations  with  respect  to  time,  e.g. -the  most  current 
observations  will  have  more  weight  than  older  ones  [34,  64,  89).  This  thesis  effort  fociisc's  on  the  use 
of  constant  weighting  for  all  software  failures;  however,  the  implementation  design  must  be  such 
that  a  weighting  scheme  based  on  severity  levels  can  be  implemented  in  the  future. 


€.2  Rcqniremenis  Analysts 

Structured  analysis  techniques  were  used  to  determine  requirements.  The  initial  requirements 
definition  was  then  expres.seil  as  a  requirements  specification  through  the  use  of  a  context  <liagram 
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SYSTERR 


Figure  C.l.  Level  0  Context  Diagram  for  SRSAS 


and  data  flow  diagrams  (DFDs)  [82].  Although  a  classical  requirements  analysis  approach  was  used, 
the  resulting  specification  can  be  applied  to  either  structured  design  (with  functional  decomposition) 
or  Object  Oriented  Design  (OOD)  [13],  The  initial  context  diagram  for  the  Software  Reliability 
Statistical  Analysis  System  (SRSAS)  is  shown  in  Figure  C.l.  The  HQ  AFOTEC  software  maturity 
data  base,  SYSTERR,  provides  the  initial  input  into  the  SRSAS,  with  additional  test  times  and 
durations  input  as  neces.sary.  The  output  is  the  Software-Reliability -Statistics,  in  terms  of  failures, 
failure  intensities,  and  confidence  intervals,  that  are  necessary  to  as.sess  the  candidate  models.  The 
SRSAS  is  further  refined  in  the  breakout  of  lower  level  DFDs. 


C.  '2.1  Lfvtl  I  DFD.  The  Level  I  DFD  is  shown  in  Figure  C.2.  The  SRS.AS  was  decom¬ 
posed  into  the  four  functions  Reduce-Data,  Assign-Times.  Determine-E.xecution-Time-Data,  and 
Determine-Logarithmic-Time-Data.  Reduce-Data  uses  the  incoming  SYSTERR  data  base  to  gen¬ 
erate  a  reduced  set  of  failure  count  data.  Assign -Times  uses  this  data  output  (as  well  as  any 
additional  time  duration  data  from  the  user)  to  assign  execution  times  to  failures  and  calcti- 
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Figure  ( '.2.  Level  1  DFD  for  SHSAS 


late  lime  statistics,  such  as  total  test  time.  This  information  is  used  by  the  other  two  functions 
Determine. Execution_Time-Data  and  Determine.Logarithmic.Time.Data.  The  functions  Deter- 
niinc-Execution.Time.Data  and  Delermine.Logarithmic-Time.Data  are  ba.sed  on  examples  and 
ecpiations  in  Musa  et  al.;  however,  no  further  decomposition  of  the  functions  Reduce.Data  and 
A.ssign.Times  is  possible  without  making  design  decisions. 


C.2.1.1  Level  2  DFD:  Deiermtne.Eiccutton-Timi.Dala.  The  Level  2  DFD  for  Deter- 
mine-Execution.Time.Data  is  shown  in  Figure  C.3.  The  time  and  failure  information  is  taken 
and  applied  against  the  program  structure  identified  in  Musa  el  al.  for  a  tabular  software  re- 
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Reduced.] 


Figure  C.3.  Level  2  DP'D  for  Detennine.Executioii.Tinie-Data 

liability  program  [t)-1:o88-589].  The  output  of  this  then  goes  to  the  user  in  the  form  of  Soft- 
ware.Reliability.Statistics. 

C.2.1.2  Lfvtl  2  DFD:  Diitrmine.Logartihmtc.Timt.Dala.  The  Level  2  DFD  for  De¬ 
termine. Logarithmic.Time. Data  is  shown  in  Figure  C.4.  As  with  the  Determine. Execution-Time. Data 
module,  time  and  failure  information  is  taken  and  applied  against  the  program  structure  identified 
in  Musa  et  al.  for  a  tabular  software  reliability  program  {64;588-589].  The  output  of  this  also  goes 
to  the  u.ser  in  the  form  of  Soft  ware. Reliability  .Statistics. 

C.:i  Rtgiiiroixvls  SiHctficalioii 

Specification  of  the  initial  ro<|uiremonts  was  performed  based  on  the  Level  0  Context  Dia¬ 
gram.  and  the  lower  level  DFDs,  which  defined  the  objects  and  functions  of  the  system.  These 
specifications  formed  the  ba-seline  for  the  softw'are  design. 


krhalvbiy. 
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Figure  C'.4.  Level  ‘2  DFD  for  Determine-Logaritlimic.Tinie.Data 


€-4  Software  Design 

The  software  design  effort  was  at  two  levels:  high-level  design  effort  (w'hich  included  transition 
front  structured  analysis  to  OOD),  and  abstract  data  type  (ADT)  selection  and  low-level  design 
effort.  The  waterfountain  approach  allowed  a  chance  to  revisit  the  different  design  levels,  as  well  as 
the  initial  analysis,  throughout  the  design  proces.s  (84].  The  di.sctission  in  the  following  (taragraphs 
reflects  that  iterative  nature,  aiul  will  present  the  design  effort. 

The  front  end  of  the  analysis  was  based  on  structured  analysis  techniques,  however,  there 
is  a  need  to  establish  historical  data  from  previous  software  reliability  analysis  to  enable  future 
validation  efforts  of  other  .software  reliability  models  [26:200],  Toward  this  end.  the  concept  of 
data  persistence  will  be  encorporated  into  the  design  effort,  specifically  in  the  development  of  data 
stores  for  the  different  functions  to  ust.'.  In  order  to  optimize  the  design  and  implementation  of  the 
data  base  and  the  accompanying  software,  a  selection  of  the  most  appropriate  data  model  must  be 
made.  The  model  itself  is  simply  a  collection  of  conceptual  tools  that  can  be  used  to  describe  the 
actual  data,  data  semantics,  data  relationships,  and  existing  consistency  constraints  between  data 
[19:6]. 
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While  tliere  have  been  many  data  models  proposed  and  implemented  for  databases,  they  fall 
into  four  basic  categories;  physical  data  models;  record-based  logical  models;  object-based  logical 
models:  and  object-oriented  models  [49:6], [97:7].  Of  the  first  three,  object-ba.sed  logical  models  are 
the  best  suited  for  the  logical  and  external  schemas  of  describing  data  at  both  the  conceptual  and 
view  levels  [49:6].  The  physical  design  of  the  data  base  would  then  be  done  in  a  relational  model 
for  the  internal  schema.  Object-oriented  models  include  the  aspects  of  object-bcised  models  (object 
identity  and  type  hierarchy),  as  well  as  data  abstraction  and  user  defined  operations  [97:92].  This 
makes  the  object-oriented  models  better  suited  for  .schema  description  at  all  three  levels  (internal, 
logical,  and  external);  however,  the  Clipper  programming  language  supports  a  more  relational 
implementation  of  the  data  base  at  the  internal  schema  level  [66:3-7].  This  requires  at  least  an 
object-based  model,  if  not  an  object-oriented  model.  In  support  of  this,  a  transformation  to  the 
method  of  OOD  was  done  using  the  following  steps  [12:17]; 

•  Identify  objects  and  their  attributes  from  all  sources  and  destinations  of  data  as  well  as  data 
stores. 

•  blentify  all  operations  suffered  by  and  required  of  each  object. 

•  Establish  visibility  among  objects. 

•  Establish  interfaces  of  objects. 

C.-/./  hUvtificdtioii  of  Objfcts  and  Thttr  AHribuiis.  The  initial  Level  1  DFD  was  revised, 
taking  into  account  the  design  decision  to  incorporate  data  persistence  (Figure  C.o).  From  this 
final  DFD.  the  following  objects  and  attributes  were  identified: 

•  SYSTERR  Data,  with  attributes  Date.  Severity  Level,  Software  Problem  Report  Number, 
and  Description. 

•  Failure-Count,  with  attributes  of  Date  and  Number  of  Failures  for  each  Severity  Level. 


•  Test.Time.  with  at  tributes  of  Test. Duration.  Time.Grouping, 


Figure  C.5.  Revised  Level  1  DFD  for  SRSAS 


•  Failure-Time,  with  attributes  of  Dale,  Local.Time.Of.Ocrurrence.  T*^t. Duration,  Total-Time, 
and  Total-Time-Of.Occurrence. 

•  Software -Reliability -Statistics,  with  attributes  of  Time,  .Mumber-Of.Failures-Experienced, 
Number.Of-Failures-E.xpected,  and  Failure-Intensity. 


C.4-2  Operations  Suffered  by  and  Required  of  Each  Object.  Operations  that  identify  the 


behavior  of  each  object  were  identified  as  follows: 


•  SYSTERR  Data:  none. 


•  Failure.Count:  add  up  data  common  to  the  same  Date,  sort  the  data  in  chronological  order. 

•  Test-Time:  none. 

•  Failure-Time:  determine  Local-Time-Of.Occurrence.  Test-Duration.  Total-Time,  and  To- 
tal-Time-Of-Occurrence. 

•  Software-Reliability -Statistics:  determine  Number.Of-Failures-Experienced.  FailureJntensity, 
anti  Number-Of-Failures-Expecled. 

Operations  that  are  required  of  each  object  were  itlentified  as  follows: 

•  SYSTERR  Data:  provide  Date  and  Severity  Level. 

•  Failure-Count:  provide  Date  and  TotaLFailures.to-Date. 

•  Test.Time:  provide  Test. Duration  and  Time.Grouping. 

•  Failure.Time:  provide  LocaLTime.Of.Occurrence.  Test. Duration.  Total.Time.Of.Occurrence, 
and  Total-Time. 

•  Software.ReliabilityJStatistics:  provide  Nuiiiber-Of.Failures-Experienced,  FailureJntensity, 
and  Number-Of-Failures-Expected. 

C.4.3  Esiahhsh  Vtstbiliiy  Aviovg  Objfds.  The  visibility  among  objects  is  ba.sed  on  the  re¬ 
lationships  between  the  databases,  and  is  shown  in  Figure  C.t).  The  module  diagram  is  the  basis 
for  transformation  of  the  design  information  into  an  implementation  (iti  this  case.  Clipper  code). 
The  objects  come  directly  from  those  identified  above.  As  naming  conventions  for  an  MS-DOS  en¬ 
vironment  are  limited  to  eight  characters,  with  Clipper  supplying  the  .PRG  extension,  and  Clipper 
is  more  functional  than  object-oriented,  the  objects  were  broken  out  into  program  modules  and 
databases  with  a  basic  naming  convention  (see  Table  C.l). 
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Figure  C.(5.  Visibility  Among  SRSAS  Objects 

Esidhii.'ili  Ivterjacts  of  Objects.  The  interfaces  of  the  oli  jects  are  normally  writ  ten  as 
pari  of  the  code,  such  as  the  package  specification  in  Ada.  1’his  also  was  performed  for  flipper, 
which  is  a  more  functional  than  object-oriented  programming  language  (see  Appendix  D). 

C.o  Dftermnif  Need  for  Abstract  Data  Type 

Basetl  on  the  requirements  provided  and  the  availability  of  the  Clipper  programming  envi¬ 
ronment,  a  data  base  implementation  was  chosen  over  the  development  and  implementation  of  a 
specialized  ADT.  This  further  simplified  the  low-level  tlesign  process,  as  the  software  was  required 
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Table  C.l.  Listing  of  Objects  and  Implonientation  Name 


Object 

Database 

FAILURE-COUNT 

COUNT.  PRC 

COUNT. DBF 

FAILURE-TIME 

SRTIME.PRG 

SRTBUILD.PRG 

TIME. DBF 

SRTDATE.PRG 

TIME. DBF 

SRTBl.PRG 

BIDATA.DBF 

SRI'EST.PRG 

COUNT. DBF 

TIME. DBF 

coil  NT.  DBF 

TIME. DBF 

TIMEDTE.DBF 

SOFTWARE-RELIABILITY. 

STATISTICS 

SREXEC.PRG 

TIME. DBF 

SRLOG.PRG 

TIME. DBF 

only  to  manipulate  the  data  in  the  database,  and  not  perform  the  database  implementation  it¬ 
self.  Thus,  no  specific  ADT  was  necessary  or  would  provide  additional  capabilities  for  software 
development . 
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Appendix  D.  Candidate  Software  Reliability  Model  Implementation  Code 


This  appendix  contains  the  code  that  was  developed  in  order  to  perform  evaluation  of  the 
candidate  software  reliability  models. 


D.]  Software  Reliability  Siatistical  Analysis  Software  (SRSAS) 

m*^l^l**^H^^^^I>^:,^**************>^*m*  ***************  ****t**************’^********  ****** 
♦  ♦ 

*  Title  ;  Software  Reliability  Statistical  Analysis  System  (SRSAS)  ♦ 

*  Version  :  3.3  * 

*  Date  :  15  Oct  91  ♦ 

*  Author  :  Capt  Joseph  J .  Stanko  ♦ 

*  Security  :  Unclassified  * 

*  Purpose  :  This  program  does  three  things;  ♦ 

*  1.)  Calculate  initial  statistics  from  existing  SYSTERR  * 

*  software  maturity  database  for  use  in  software  reliability  * 

*  model  evaluation  * 

*  2.)  Generate  a  database  of  failure  times  based  on  average  test  * 

*  times,  actual  test  time  times,  or  estimated  test  times.  * 

*  3.)  Perform  calculations  on  data  to  determined  goodness-of-f it  ♦ 

*  for  each  candidate  software  reliab'’ity  model.  * 

*  * 

*  Theory  :  1 . )  The  prograun  checks  for  the  ex.,.stence  of  a  summary  database  ♦ 

*  and  if  one  does  not  exist,  one  is  constructed  solely  from  * 

*  the  SYSTERR  fields  of  the  software  maturity  database  ♦ 

*  2.)  (Text ,  the  program  prompts  for  data  not  in  the  SYSTERR  * 

*  database  (such  as  total  test  time  or  test  durations)  and  ♦ 

*  generates  a  database  of  failure  t^mes.  ♦ 

*  3.)  Finally,  the  program  takes  the  failure  time  information  * 

*  and  calculates  estimates  of  model  parameters  and  their  * 

*  confidence  intervals.  Outputs  are  given  (in  tabular  form)  * 

*  of  actual  and  estimated  data. 

*  NOTE : 

*  This  is  the  initial  transition  from  existing  SYSTERR  databases 

*  to  the  software  reliability  database  for  the  Reliability 

*  Analysis  System  (RAS).  Future  SYSTERR  database  configuration 

*  based  on  the  1  Oct  1990  AFOTECP  800-2  Vol  6  will  have 

*  fields  that  will  be  handled  by  RAS  itself  as  an  integrated 

*  package. 

* 

*  Database  :  This  progreun  uses  three  databases; 

* 

*  SYSTERR. DBF  -  There  were  several  different  "versions"  of  this 

*  database  done  by  each  test  te2un.  The  following 

*  are  the  fields  found  common  to  each  that  might 

*  be  useful  for  software  reliability  analysis; 

* 

*  Name  Type  Length  Decimal  Description 

«  - —  —  —  —  —  —  - —  — - - - ^ 

*  DATE  Date  Date  of  occurrence  of  failure  ♦ 

*  CPCI  C  10  CPCI  associated  with  failure  * 

*  SEV_C0DE  C  1  Severity  Code  (1-5)  of  failure  * 

*  DATE_FIX  Date  Date  failure  fixed  * 

*  TITLE  C  42  Description  of  failure  ♦ 

*  PROB.NUM  C  10  Software  Problem  Number  (SPR)  ♦ 

*  * 

*  While  this  data  is  available  from  the  existing  SYSTERR  database  * 

*  it  does  not  include  the  time  values  necessary  for  reliability  * 

*  evaluation.  This  data  must  be  prompted  for  from  the  user.  * 

*  ♦ 

*  COUNT. DBF  -  This  is  an  intermediate  summary  database  of  * 

*  dates  and  number  of  failures;  * 

*  * 
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*  *  *  «  «#»#*«  *  *  «  #  «««« 


Type  Length  Decimal 


Description 


CAL_DATE  E 

SEV_C0DE_1 

SEV  C0DE_2 

SEV_C0DE_3 

SEV_C0DE_4 

SEV_C0DE_5 

N0_SEV_C0DE 

TOT.NUM 

TOTAL 

TIME. DBF 


ate  Date  ot  occurrence  of  failure  * 

N  4  #  of  Severity  Code  1  failures  * 

N  4  #  of  Severity  Code  2  failures  * 

N  4  #  of  Severity  Code  3  failures  * 

N  4  #  of  Severity  Code  4  failures  ♦ 

N  4  #  of  Severity  Code  5  failures  ♦ 

N  4  #  of  failures  not  coded  ♦ 

N  4  Total  Number  for  this  date  ♦ 

N  4  Overal  total  of  failures  ♦ 

♦ 

-  This  is  a  final  database  of  dates  and  estimated  * 
failure  times  cind  test  durations:  * 


Length  Decimal 


Description 


CAL.DATE 

L_TIME_OCC 

TEST.DUR 

TTIME.OCC 


TOTAL 


Date  of  occurrence  of  failure 
"Local"  time  of  failure  occur 
(wrt  to  start  of  that  test) 
Duration  of  test  for  that  day 
"Total"  time  of  failure  occur 
(wrt  to  all  total  test  time) 
Total  failures  to  that  point 


*  Nodules  :  Calls  the  following  modules  for  operation:  * 

*  SRCOUNT.PRG  -  Initializes  the  database  COUNT. DBF,  reduces  ♦ 

*  the  SYSTERR.DBF  entries  into  a  count  summary  ♦ 

*  form,  and  puts  in  ascending  chronological  order.* 

*  SRPRINT.PRG  -  Prints  the  COUNT. DBF.  ♦ 

*  SRTIME.PRG  -  Initializes  and  generates  the  database  TIME. DBF.* 

*  SREXEC.PRG  -  Perform  calculations  on  the  TIME. DBF  data  with  * 

*  Musa  Execution  Time  Model.  * 

*  SRLOG.PRG  -  Perform  calculations  on  the  TIME. DBF  data  with  * 

*  Musa-Okumoto  Logarithmic  Erection  Time  Model.  * 

*  * 


clear  screen 


*  Variable  Section: 

option  =  "C"  memvar  for  main  menu 


*  Set  Section; 

set  decimal  to  9  set  decimal  length  beyond  default  (2) 

jf; - - - 

*  Main  Loop: 

do  while  upper (option)  <>  "X" 
set  color  to  w+/b,g/n 
ffl  0,0  cle2a' 

<D  3,12  say  "Software  Reliability  Statistical  Analysis  System  (SRSAS)" 

®  4,12  say  "  Version  3.3,  Oct  1991" 

ffl  6,20  say  "C  -  Create  Count  Data  Base" 

ffl  8,20  say  "P  -  Print  Count  Data  Base" 

ffl  10,20  say  "T  -  Create  Time  Data  Base" 

ffl  12,20  say  "E  -  Execution  Time  Model" 

ffl  14,20  say  "L  -  Logarithmic  Poisson  Execution  Time  Model" 

ffl  16,20  say  "X  -  Exit" 

ffl  20,20  say  "Please  Enter  Option:"; 

get  option  picture  "«K  !"  valid(option$"CPTELX") 

read 

do  case  tt  Call  sr  prograuns  based  on  menu  input : 

case  upper(option)="C" 
do  srcount 

case  upper(option)="P" 
do  srprint 

case  upper(option)="T" 
do  srtime 

case  upper(option)="E" 


do  srexec 

case  upper (opt ion )="L" 
do  srlog 

case  upper(option)="X" 

fi  24,20  say  "Exiting  This  Program  ..." 

otherwise  &&  standard  exception  handling 

®  23,20  say  "Invalid  Entry  -  Please  Use  C,P,T,E,L,  or  X" 
endcase 

enddo 

1)1 - 

clear  all 
clear  screen 
quit 
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D.2  Software  Reliability  COUNT. DBF  Module 


♦  ♦ 

♦  Title  :  Software  Reliability  COUNT. DBF  Module  (SRCOUNT.PRG)  * 

♦  Version  ;  3.3  * 

♦  Date  :  2  Oct  91  * 

♦  Author  :  Capt  Joseph  J.  Stanko  * 

♦  Security  ;  Unclassified  ♦ 

♦  Purpose  :  This  program:  * 

♦  1.)  Calculates  initial  summary  statistics  from  the  SYSTERR  ♦ 

♦  softwaire  maturity  databases  for  use  in  software  reliability  * 

♦  model  evaluation.  ♦ 

♦  4c 

♦  Theory  :  One  pass  module.  The  program  checks  for  the  existence  of  a  * 

♦  summary  database  and  if  one  does  not  exist,  one  is  constructed  * 

♦  solely  from  the  SYSTERR  fields  of  the  software  maturity  * 

♦  database.  * 

♦  ♦ 

*  Database  :  This  program  uses  two  databases:  * 

*  * 

♦  SYSTERR. DBF  -  There  were  several  different  "versions"  of  this  * 

♦  database  done  by  each  test  team.  The  following  * 

♦  are  the  fields  found  common  to  each  that  might  * 

♦  be  useful  for  software  reliability  amalysis:  * 


* 

Naune  Type 

Length 

Decimal  Description 

♦ 

♦ 

4c 

DATE  Date 

Date  of  occurrence  of  failure 

* 

4c 

CPCI 

C 

10 

CPCI  associated  with  failure 

4c 

4c 

SEV.CODE 

C 

1 

Severity  Code  (1-5)  of  failure 

* 

* 

DATE  FIX  Date 

Date  failure  fixed 

TITLE 

C 

42 

Description  of  failure 

* 

* 

PROB.NUM 

C 

10 

Software  Problem  Number  (SPR) 

* 

* 

While  this 

data 

is  available  from  the  existing  SYSTERR  database 

* 

it  does  not 

include  the 

time  values  necessary  for  reliability 

4t 

♦ 

evaluation. 

This  data  must  be  prompted  for  from  the  user. 

* 

4c 

COUNT. DBF 

- 

This  is 

an  intermediate  summary  database  of 

* 

♦ 

dates  and  number  of  failures: 

* 

Name 

Type 

Length 

Decimal  Description 

* 

* 

CAL.PATE 

Date 

Date  of  occurrence  of  failure 

* 

SEV  CODE  1 

N 

4 

#  of  Severity  Code  1  failures 

* 

SEV  CODE  2 

N 

4 

#  of  Severity  Code  2  failures 

4c 

SEV.CODE  3 

N 

4 

#  of  Severity  Code  3  failures 

* 

4c 

SEV  CODE  4 

N 

4 

#  of  Severity  Code  4  failures 

4: 

SEV  CODE  5 

N 

4 

#  of  Severity  Code  5  failures 

* 

NO.SEV  CODE  N 

4 

#  of  failures  not  coded 

* 

4c 

TOT.NUM 

N 

4 

Total  Number  for  this  date 

* 

4c 

TOTAL 

N 

4 

Overal  total  of  failures 

* 

4c 

Modules  :  None . 

4c 

4c«4c4c4c4c4c4c4c*4c*4c4c4i4c*4c4c4i«J4c4  4c*4c**«4c*«*«4c4c4c««*4c**4c***«**4c**4c*4>»4c*4c«**4c«4cc4«4c4c«*«*4>i^«4c* 

4c 

Check  for  COUNT. DBF  or  create  if 

:  it  does  not  exist: 

if  .not.  file ("COUNT. DBF") 

C  23,20  say  "Building  COUNT. DBF  Database  ..." 
create  template 
use  template 
append  blank 

replace  field.name  with  "Cal_Date", 
append  blank 

replace  field_name  with  "Sev_Code_l" , 
field_len  with  4 
append  blank 

replace  field.name  with  "Sev_Code_2", 
field.len  with  4 
append  bl2mk 

replace  field.name  with  "Sev_Code_3" , 


field.type  with  "DATE" 
field.type  with  "N", 

field.type  with  "N", 

field.type  with  "N", 
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f ield_len 
append  blank 
replace  field.name 
f ield_len 
append  bl2aik 
replace  field_n2une 
f ield_len 
append  blank 
replace  field_neune 
f ield_len 
append  bleink 
replace  field_naine 
f ield.len 
append  blank 
replace  field_neune 
f ield_len 

go  top 
close  all 

file  =  "COUNT. DBF" 
create  ftfile.  from 
erase  template.dbf 


*  Now  reduce  the 


with  4 


with  "Sev_Code_4" 
with  4 

with  "Sev_Code_5" 
with  4 

with  "No_Sew_Code 
with  4 

with  "Tot.Hnm", 
with  4 

with  "Total", 
with  4 


field_type  with 
field_type  with 
field_type  with 
field_type  with 
field_type  with 


"N" 


"N" 


"N" 


template 


SYSTERR.DBF  data  into  the  COUNT. DBF  database; 


use  COUNT  alias  COUNT 
sgIgcI*  2 

use  SYSTERR  alias  MATURITY 

store  0  to  mtot 
store  0  to  mscl 
store  0  to  msc2 
store  0  to  msc3 
store  0  to  msc4 
store  0  to  msc6 
store  0  to  mnsc 
store  DATE  to  mdate 


ftk  Aliases  sure  do  help  dis2unbiguate  veirs: 

At  Initialize  counts  for  total  and  all 
Aft  severity  codes  (sc’s  1-5) 

AA  Just  in  case  some  are  "no  severity  code" 


4  0,0  clear 

C  5,20  say  "Tabulating  Count  Data 


do  while  .not.  eof()  AA 

mtot  =  mtot  +1  AA 

store  SEV_C0DE  to  msevcode  AA 
do  case  Aft 


Since  each  entry  in  the  SYSTERR  database 
is  a  single  and  sepetrate  failure,  all 
must  be  added  up  by  date  with  summary 
information  on  all  severity  codes 


case  msevcode  =  "1" 
mscl  =  mscl  +  1 
case  msevcode  =  "2" 
msc2  =  msc2  +  1 
case  msevcode  =  "3" 
msc3  =  msc3  +  1 
case  msevcode  =  "4" 
msc4  =  msc4  +  1 
case  msevcode  =  "5" 
msc5  =  msc5  +  1 
otherwise 

mnsc  =  mnsc  +  1 
endcase 
skip 


if  DATE  <>  mdate 
select  COUNT 


Aft  Check  to  see  if  we've  moved  to  another  date 
AA  If  we  have,  save  off  the  sufflm2try  data 


append  bleink 

replace  CAL_DATE  with  mdate 
replace  SEV_C0DE_1  with  mscl 
replace  SEV_C0DE_2  with  msc2 
replace  SEV_C0DE_3  with  msc3 
replace  SEV_C0DE_4  with  msc4 
replace  SEV_C0DE_5  with  mscS 
replace  N0_SEV_C0DE  with  mnsc 
replace  T0T_NUM  with  mtot 
replace  TOTAL  with  0 


store  0  to  mscl 
store  0  to  msc2 
store  0  to  msc3 
store  0  to  msc4 
store  0  to  msc5 


AA  reinitialize  the  summary  variables 


Do 


store  0  to  mnsc 
store  0  to  mtot 

select  MATURITY  tk  and  do  it  again  for  the  new  date 

store  DATE  to  mdate 
end  if 
enddo 

^ - 

♦  Since  many  of  the  entries  in  the  SYSTERR  database  were  not 
in  straight  chronological  order,  the  data  needs  to  be  sorted 

♦  and  then  compressed  so  that  only  one  entry  exists  for  any  given  date 

®  7,20  say  "Sorting  the  Tabulated  Data  ..." 
select  COUNT 

sort  on  CAL_DATE  to  tempi 
select  3 
use  tempi 

®  9,20  say  "Compressing  Tabulated  Data  ..." 

store  1  to  rec_num 

store  CAL_DATE  to  mdate 

store  SEV_C0DE_1  to  mscl 

store  SEV_C0DE_2  to  msc2 

store  SEV_C0DE_3  to  msc3 

store  SEV_C0DE_4  to  msc4 

store  SEV_C0DE_5  to  mscS 

store  N0_SEV_C0DE  to  mnsc 

store  T0T_NUM  to  mtot 

skip 

do  while  .not.  eof() 
rec_num  =  rec_num  +  1 
if  CAL_DATE  =  mdate 

mscl  =  SEV  CODE  1  +  mscl 
msc2  =  SEV.CODE  2  +  msc2 
msc3  =  SEV_C0DE_3  +  msc3 
msc4  =  SEV_C0DE_4  +  msc4 
mscS  =  SEV_C0DE_S  +  msc5 
mnsc  =  K0_SEV_C0DE  +  mnsc 
mtot  =  T0T_NUM  +  mtot 
replace  SEV_C0DE_1  with  mscl 
replace  SEV_C0DE_2  with  msc2 
replace  SEV_C0DE_3  with  msc3 
replace  SEV_C0DE_4  with  msc4 
replace  SEV_C0DE_5  with  mscS 
replace  N0_SEV_C0DE  with  mnsc 
replace  T0T_NUM  with  mtot 
goto  rec_num  -  1 
delete 

goto  rec_num 
endif 

store  CAL_DATE  to  mdate 
store  SEV_C0DE_1  to  mscl 
store  SEV_C0DE_2  to  msc2 
store  SEV_C0DE_3  to  msc3 
store  SEV_C0DE_4  to  msc4 
store  SEV_C0DE_5  to  mscS 
store  N0_SEV_C0DE  to  mnsc 
store  T0T_NUM  to  mtot 
skip 
enddo 

♦  - 

*  Now  siun  the  totals  and  include  in  the  COUNT. DBF: 
go  top 

mtot  =  0 

do  while  .not.  eof() 

store  T0T_NUH  +  mtot  to  mtot 
replace  TOTAL  with  mtot 
skip 
enddo 

♦  - — 

*  And  close  up  shop: 

pack 
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close  all 

erase  COURT. DBF 

rename  tempi. dbf  to  COUNT. DBF 

qXS6 

8  23,20  say  "COUNT. DBF  Database  Already  Exists 
wait  "Hit  any  key  to  continue  ..." 
endif 


return  AA  to  SRSAS  main  menu 

^^^^^^^^^m^i^*^:^^^^^******^^*********************  *******  ************************* 
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D.3  Software  Keltabthiy  Print  Module 


*  * 

*  Title  :  Software  Reliability  Print  Module  (SRPRIMT.PRG)  * 

♦  Version  ;  3.3  ♦ 

*  Date  :  2  Oct  91  ♦ 

*  Author  ;  Capt  Joseph  J.  Stcinko  4^ 

*  Security  :  Unclassified  ♦ 

♦  Purpose  ;  This  program:  ♦ 

♦  Prints  the  contents  of  the  COUNT. DBF  to  either  a  screen,  ♦ 

♦  printer,  or  data  file.  ♦ 

•  Currently  only  the  COUNT. DBF  is  useful  for  output — the  TIME. DBF  * 

*  has  one  entry  for  each  failure  recorded,  auid  that  would  use  ♦ 

♦  a  lot  of  paper  to  print.  However,  it  would  be  simple  to  modify  ♦ 

♦  this  program  to  print  the  TIME. DBF  information  to  a  file  if  * 

*  needed.  ♦ 

*  ♦ 

*  Theory  :  User  is  given  option  of  where  to  print  the  COUNT. DBF  database.  ♦ 

*  It’s  a  one  pass,  with  default  values  initialized  for  screen  ♦ 

*  output  (saves  on  paper!).  * 

*  * 

»  Database  :  This  program  uses  one  database;  * 

♦  * 

*  COUNT. DBF  -  This  is  an  intermediate  summary  database  of  * 


* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 


dates  and  number  of  failures: 

Name  Type  Length  Decimal  Description 


* 

* 

* 

m 


CAL_DATE  Date 
SEV_C0DE_1  N  4 

SEV_C0DE_2  N  4 

SEV_C0DE_3  N  4 

SEV_C0DE_4  N  4 

SEV.CODE.S  N  4 

N0_SEV_C0DE  N  4 

TOT.NUM  N  4 

TOTAL  N  4 


Date  of  occurrence  of  failure  * 

#  of  Severity  Code  1  failures  * 
tt  of  Severity  Code  2  failures  * 

#  of  Severity  Code  3  failures  * 

#  of  Severity  Code  4  failures  * 

#  of  Severity  Code  5  failures  • 

#  of  failures  not  coded  * 

Total  Number  for  this  date  * 
Overall  total  of  failures  * 


Modules 


N/A 


******<*******************««*  ******«*#*4>»  ***«*«»  ********4>********4i******  **«•*#** 


♦  Variable  Section: 

prvar  =  "S"  Aft  Variable  for  print  option 

store  1  to  LOC  ftft  Line  of  Code — used  for  printing  information 

store  "  "  to  dbname  ftA  Name  of  database  for  output  header 

t - 

♦  First,  see  if  COUNT. DBF  exists: 
if  file ("COUNT. DBF") 

♦  — — - — - —  —  — - -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  - - — - 

*  If  it  does  then  do  print,  etc. 

e  23,10  say  "Print  Data  to  (S)creen,  (P)rinter,  (F)ile,  or  (R)eturn:": 
get  prvar  picture  "«K  !"  valid(prvarS"SPFR") 

read 

9  23,10  clear 

if  upper(prvar)  <>  "R"  tt  Make  sure  we  don’t  wamt  to  return  to  SRSAS 

9  0,0  clear 

use  COUNT 

replace  TOTAL  with  T0T_NUM 

9  6,20  say  "Data  Base  Neune  for  Header;"  get  dbn2une  picture  "!!!!!!!!" 

read 

if  upper(prvar)  =  "F"  Aft  Specific  parameters  for  file  output 

C  7,20  say  "Sending  Data  to  File  SRCOUNT.PRN  ..." 
set  printer  to  SRCOUNT.PRN 
set  device  to  print 

pagelength  =  4000  M  Pagelength  large  so  header  info  used  once 

elseif  upper(prvar)  =  "P"  Aft  Specific  parameters  for  printer  output 
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Q  7,20  say  "Printing  Results  ..." 
set  device  to  print 
pagelength  =  56 

else  ftfe  Specific  pairaneters  for  screen  output 

clear 

pagelength  =  20 
endif 

do  while  .not.  eof() 

if  LOG  =1  Output  header  information 

9  LOG, 20  say  "Database  for  " 

9  LOG, 33  say  dbname 
store  LOG+2  to  LOG 
«  LOG.l  say  "Date" 

9  LOG, 10  say  "#  1" 

9  LOG, 15  say  "#  2" 

9  LOG, 20  say  "#  3" 

«  LOG, 25  say  "#  4" 

9  LOG, 30  say  "#  5" 

9  LOG, 35  say  "NSG" 

9  LOG. 40  say  "Total" 

0  LOG, 50  say  "Gum  Total" 

C  LOG+1.1  say  " - 

store  LOG  +  2  to  LOG 
endif 

9  LOG.l  say  GAL.DATE  Aft  Output  summary  database  information 

«  LOG, 10  say  SEV  GODE.l 

a  LOG, 15  say  SEV_G0DE_2 

9  LOG, 20  say  SEV  G0DE.3 

<1  LOG, 25  say  SEV  G0DE_4 

<1  LOG, 30  say  SEV_G0DE_5 

9  LOG. 35  say  N0_SEV_G0DE 

9  LOG. 40  say  TOT.NUM 

9  LOG, SO  say  TOTAL 

store  LOG  +  1  to  LOG 

if  LOG  =  pagelength  ftft  Reset  for  beginning  of  new  page 

store  1  to  LOG 
if  upper(prvar)  =  "S" 

wait  "Hit  any  key  to  continue  ..." 
clear 
endif 
endif 

skip 

enddo 

if  upper (prvar)  =  "P"  ftft  Reset  all  parameters  for  printer  and  file 

eject 

set  device  to  screen 
elseif  upper(prvar)  =  "F" 
set  device  to  screen 

set  printer  to  ftft  reset  if  used  for  file  output 

else  ftft  If  screen  output,  pause  for  one  last  look 

wait  "Hit  any  key  to  continue  ..." 
endif 

close  all 
endif 

4, - 

*  If  GOUNT.DBF  did  not  exist: 
dsG 

C  23,10  say  "GOUHT.DBF  Does  (tot  Exist." 
wait  "Hit  any  key  to  continue  ..." 
endif 


return  ftft  to  SRSAS.PRG  main  menu 

*0:^i^iim*0******  ***************************************************************** 
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**  *  *  *  *  *  *  **  ******  ***  ***  ****  *  **  ***********  *  *  *  *  ******  *****  *  ***  *  ** 


D.4  Sofiwart  Rehabtlity  TIME. DBF  Module 

♦  * 

♦  Title  :  Softsare  Reliability  TIME. DBF  Module  (SRTIME.PRG)  * 

♦  Version  :  3.3  * 

♦  Date  :  2  Oct  91  * 

♦  Author  :  Capt  Joseph  J.  Stanko  * 

♦  Security  :  Unclassified  * 

♦  Purpose  :  This  program:  * 

Creates  a  TIME. DBF  data  base  if  needed. 

Determines  the  initial  time  statistics  from  the  summary 
COUNT. DBF  and  either  average  test  durations,  actual  test 
durations,  or  estimated  test  durations. 


Theory 


Database 


This  module  allows  the  user  to  create  the  time  database,  if  it 
does  not  exist,  from  both  COUNT. DBF  and  other  user/file  input 
information. 

This  module  uses  two  databases  (see  note  below) : 


COUNT. DBF 


This  is  an  intermediate  summary  database  of 
dates  and  number  of  failures: 


Type  Length  Decimal 


Description 


CAL.DATE  D 
SEV_C0DE_1 
SEV_CODE  2 
SEV.CODE  3 
SEV  CODE  4 
SEV_C0DE_5 
NO_SEV_CODE 
TOT.NUM 
TOTAL 

TIME. DBF 


Date  of  occurrence  of  failure  ♦ 

#  of  Severity  Code  1  failures  • 

#  of  Severity  Code  2  failures  * 

#  of  Severity  Code  3  failures  * 

#  of  Severity  Code  4  failures  * 

#  of  Severity  Code  5  failures  ♦ 

#  of  failures  not  coded  • 

Total  Number  for  this  date  ♦ 

Overal  total  of  failures  • 


This  is  a  final  database  of  dates  and  estimated  * 
failure  times  and  test  durations:  * 


failure  times  and  test  durations: 

Type  Length  Decimal  Description 


CAL.DATE 

L.TIME.OCC 

TEST.DUR 

T.TIME.OCC 

TOTAL 


Date  of  occurrence  of  failure  * 
"Local"  time  of  failure  occur  ♦ 
(wrt  to  start  of  that  test)  * 
Duration  of  test  for  that  day  * 
"Total"  time  of  failure  occur  * 
(wrt  to  all  total  test  time)  * 
Total  failures  to  that  point  * 


Note:  An  additional  database  is  used  to  input  the  B-IB  flight 
test  data: 

BIDATA.DBF  -  This  is  a  summary  database  of  B-IB  flight  test 
hours  and  dates : 


Type  Length  Decimal 


Description 


DATE 

FLT.HRS 

FLT 


Date  of  mission  flown 
Mission  duration  in  hours 
Mission  identifier 


Modules:  This  program  calls  the  following  modules: 


SRTBUILD.PRG  - 
SRTDATE.PRG 


SRTBl.PRG 

SRTEST.PRG 


Creates  the  structure  for  the  TIME. DBF  if  one 
does  not  already  exist. 

Generates  the  TIME. DBF  based  on  the  assumption 
that  failure  dates  from  COUNT. DBF  are  the  only 
test  dates,  and  uses  an  average  test  duration 
input  from  the  user. 

Generates  the  TIME. DBF  from  specific  B-IB 
flight  test  data  and  the  COUNT. DBF. 

Generates  the  TIME. DBF  from  estimates  of  total 
test  time  per  month  and  the  COUNT. DBF. 


clear  screen 
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*  *  *  *  «  « 


•  Variable  Section: 

timeoption  =  "C"  ftft  memvar  for  main  menu 


*  Main  Loop: 

do  while  upper(timeoption)  <>  "R" 

9  0,0  cleeur 

®  3,12  say  "Software  Reliability  Statistical  Analysis  System  (SRSAS)" 

«  4,12  say  "  Generate  TIME. DBF  Module" 

®  6,20  say  "C  -  Create  Time  Data  Base  Structure" 

®  8,20  say  "D  -  Use  Failure  Dates  ft  Average  Test  Duration  for  Data" 

C  10,20  say  "B  -  Use  B-IB  Flight  Test  Data  for  Data" 

®  12,20  say  "E  -  Use  Estimated  Test  Time  per  Month  for  Data" 

C  14,20  say  "R  -  Return" 

®  20,20  say  "Please  Enter  Option;"; 

get  timeoption  picture  "OK  !"  valid(timeoption$"CDBER") 

read 

do  case  ftft  Call  srt  progrcuns  based  on  menu  input : 

case  upper(timeoption)="C" 
do  srtbuild 

case  upper (timeoption)="D" 
do  srtdate 

case  upper (timeoption)="B" 
do  srtbl 

case  upper (timeoption) ="E" 
do  srtest 

case  upper (timeopti'-  )  'R" 

note  :  returning  to  main  program  . . . 
otherwise  ftft  standard  exception  handling 

®  23,20  say  "Invalid  Entry  -  Please  Use  C,D,B,E,  or  R" 
endcase 

enddo 


return 

*«***«*««**«**#*«**  **««««***i»***4>********»4>**««***«*«***4i««**********»*  ******** 
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D.5  Software  ReliabUtly  TIME. DBF  Build  Module 


4(  ^ 

*  Title  :  Software  Reliability  TIME. DBF  Build  Module  (SRTBUILD.PRG)  * 

♦  Version  :  3.3  * 

♦  Date  :  25  Oct  91  ♦ 

♦  Author  :  Capt  Joseph  J.  Stanko  * 

*  Security  :  Unclassified  * 

♦  Purpose  ;  This  progreun;  ♦ 

♦  Creates  the  structure  for  both  TIME. DBF  eoid  TIMEDTE.DBF.  * 

^  * 

*  Theory  :  The  program  creates  the  necessary  database  structure  if  it  does  ♦ 

*  not  already  exist.  * 

*  * 

*  Database  :  This  program  creates  the  following  database  structure:  * 

*  * 

*  TIMEDTE.DBF  -  This  is  the  DTE  version  of  TIME. DBF  to  find  the  * 

*  initial  failure  intensity  for  OTftE  calculation.  * 

*  It  has  the  identical  structure  to  TIME. DBF.  ♦ 

*  * 

*  TIME. DBF  -  This  is  a  final  database  of  dates  amd  estimated  * 

*  failure  times  amd  test  durations:  ♦ 

*  * 

*  Kaune  Type  Length  Decimal  Description  * 

♦  —  —  -  — —  _  - -  — - — -  — ^  -  -  -  -  -  - - — - - - — - - 

*  CAL_DATE  Date  Date  of  occurrence  of  failure  ♦ 

*  L_TIME_QCC  N  10  2  "Local"  time  of  failure  occur  * 

*  (wrt  to  stairt  of  that  test)  ♦ 

*  TEST_DUR  N  10  2  Duration  of  test  for  that  day  ♦ 

*  T_TIME_0CC  N  10  2  "Total"  time  of  failure  occur  * 

*  (wrt  to  all  total  test  time)  ♦ 

*  TOTAL  N  4  Total  failures  to  that  point  * 

*  * 

*  Modules  :  Calls  procedure  DBBUILD  (see  below).  ♦ 

♦  * 


******«*«4>  ***««***«**«****  *4^**««4>«*4>*****4<**4^4t**  *************************  ****** 

Store  "0"  to  dbvar 

«  23,20  say  "(D)TftE  or  (O)TftE  Database?"  ; 

get  dbvar  picture  "fiK  !"  valid(dbvar$"D0") 

read 

if  upper(dbvar)  =  "D" 

if  .not.  fileC'TIMEDTE.DBF") 

(!)  23,20  say  "Building  DTAE  TIME  Database  ...  " 

do  dbbuild 
file  =  "TIMEDTE.DBF" 
create  tfile.  from  template 
delete  file  template. dbf 
endif 

gIsq 

if  -not.  fileC'TIME.DBF") 

«  23,20  say  "Building  TIME  Database  ... 
do  dbbuild 
file  =  "TIME. DBF" 
create  ftfile.  from  template 
delete  file  template. dbf 
endif 

endif 

* - 

return  4ft  to  srtime.prg  module 


*sss  =  =  s;ss  =  =  s==s  =  ==s  =  ===  =  =  ===s  =  =  ==ss=rrssss:s=£s=rcs=s£sr  =  =  =rrr=  =  ===  =  =  =  ===  =  =  =  =  =  =  = 

*  Procedure  Section: 

******************************************************************************* 

*  * 

*  Procedure:  Database  Build  Procedure  (DBBUILD)  * 

*  Version  :  3.3  * 

*  Date  :  23  Oct  91  * 

*  Author  :  Capt  Joseph  J.  Stanko  * 

*  Security  :  Unclassified  * 

*  Purpose  :  This  module  has  the  implementation  code  for  creating  * 

*  the  structure  for  either  TIMEDTE.DBF  or  TIME. DBF  * 

*  # 

*  Database  :  This  program  creates  the  following  database  structure:  * 
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TIME. DBF 


♦ 

* 

* 


* 

This  is  a  final  database  of  dc'tes  and  estijnated  * 
failure  times  and  test  durations;  ♦ 

4^ 

Name  Type  Length  Decimal  Description  * 


CAL_DATE 

Date 

L_TIME_OCC 

N 

10 

TEST  DUR 

N 

10 

T_TIME_aCC 

N 

10 

TOTAL 

N 

4 

•  Modules  :  N/A 


Date  of  occurrence  of  failure  ♦ 
2  “Local"  time  of  failure  occur  ♦ 
(urt  to  start  of  that  test)  ♦ 
2  Duration  of  test  for  that  day  * 
2  "Total"  time  of  failure  occur  ♦ 
(wrt  to  all  total  test  time)  * 
Total  failures  to  that  point  ♦ 


* 
* 

*****♦*♦♦♦*♦+♦*♦**♦**♦***♦*♦♦♦♦♦*♦**♦♦*♦♦*♦♦*****»♦♦♦♦*♦*♦**♦*♦**♦♦*****♦***♦♦* 


procedure  dbbuild 


create  template 
use  template 
append  bleink 
replace  field_name 
append  blank 
replace  field_name 
f ield_len 
append  blank 
replace  field_name 
f ield.len 
append  blank 
replace  field_name 
f ield_len 
append  blank 
replace  field.name 
f ield_len 

go  top 
close  all 


with 

"Cal_Date" , 

f ield_type 

with 

"DATE" 

with 

"L  Time  Occur", 

f ield_type 

with 

"N", 

with 

10, 

f ield_dec 

with 

2 

with 

"Test.Dur", 

f ield_type 

with 

"H", 

with 

10, 

f ield_dec 

with 

2 

with 

"T  Time  Occur", 

f ield_type 

with 

"N", 

with 

10, 

f ield.dec 

with 

2 

with 

with 

"Total", 

4 

f ield.type 

with 

"N", 

return  tt  to  procedure  SRTBUILD 
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D.6  Software  Reliability  TIME. DBF  Date  Module 


*  * 

♦  Title  :  Software  Reliability  TIME. DBF  Date  Module  (SRTDATE.PRG)  * 

♦  Version  :  3.3  * 

♦  Date  :  2  Oct  91  * 

♦  Author  :  Capt  Joseph  J.  Stauko  * 

♦  Security  :  Unclassified  ♦ 

♦  Purpose  :  This  program:  ♦ 

♦  Generates  the  data  for  the  TIME. DBF  database  from  average  test  * 

♦  times.  ♦ 

♦  ♦ 


♦  Theory  :  The  program  generates  TIME. DBF  data  from  the  use  of  average  test* 


*  times  assumed  to  occur  ONLY  ON  THE  DATES  OF  FAILURES  as  found  ♦ 

*  in  the  COUNT. DBF  and  SYSTERR.DBF  databases.  This  assumption  is  * 

*  valid  if  testing  occurred  only  on  the  dates  that  failures  were  * 

*  identified;  however,  as  failures  cire  often  "boarded"  by  a  pcinel  * 

*  cind  recognized  at  dates  that  could  be  different  than  actual  * 

*  test  dates,  amother  option  should  be  used.  ♦ 

*  ♦ 

*  This  module  was  used  for  initial  analysis  of  data  until  more  ♦ 

*  definitive  test  times  aind  durations  were  available.  ♦ 

*  ♦ 

♦  Database  :  This  program  uses  the  following  database:  * 

♦  ♦ 

*  TIME. DBF  -  This  is  a  final  database  of  dates  and  estimated  * 

*  failure  times  cind  test  durations:  * 

*  ♦ 

*  Ncune  Type  Length  Decimal  Description  * 

*  —  — — - -  — - -  - - —  — —  4c 

*  CAL_DATE  Date  Date  of  occurrence  of  failure  * 

*  L_TIME_0CC  N  10  2  "Local"  time  of  failure  occur  ♦ 

*  (wrt  to  start  of  that  test)  ♦ 

*  TEST_DUR  N  10  2  Duration  of  test  for  that  day  * 

*  T_TIME_0CC  N  10  2  "Total"  time  of  failure  occur  ♦ 

*  (wrt  to  all  total  test  time)  * 

*  TOTAL  N  4  Total  failures  to  that  point  * 

*  * 

*  Modules  :  N/A  * 

*  * 


First,  check  to  see  if  the  TIME. DBF  exists: 
if  fileC’TIME.DBF") 

— - — - —  — — - — - -  -  - - -  - - - - - — - —  - - — 

*  Variable  Section: 

use  COUNT  alias  COUNT  t&  Database  of  failure  COUNT  data 

sgXgcI  2 

use  TIME  alias  TIME  4&  Database  of  failure  TIME  data 

select  COUNT 

store  0  to  d_offset  44  Day  offset  to  determine  total  test  time 

store  3600  to  day_val  44  Day  value  for  test  duration  (minutes) 

store  0  to  m_e  44  Total  number  of  failures 

store  0  to  max_dur  44  Max  partition  for  assigning  failure  times 

store  0  to  mtestdur  44  Local  value  for  test  duration 

store  0  to  my_tot  44  Local  total  number  of  failures 

store  0  to  num_sec  44  Number  of  seconds  from  system  clock 

store  0  to  p_offset  44  Partition  offset  for  local  failure  times 

store  CAL_DATE  to  mdate  44  Local  date  for  failure  occurrence 

store  CAL_DATE  to  strtdate  44  Starting  date  for  data  emalysis 

go  bottom 

store  CAL_DATE  to  enddate  44  Ending  date  for  data  cmalysis 
go  top 

♦  - 

*  Data  Entry  Section: 

set  confirm  on 
C  0,0  clear 

C  3,10  say  "Enter  Starting  Date  for  Data  :"  get  strtdate  picture  "99/99/99" 

®  S,10  say  "Enter  Ending  Date  for  Data  :"  get  enddate  picture  "99/99/99" 

«  7,10  say  "Enter  Daily  Test  Duration  (min):"  get  day_val  picture  "999999" 


Dll 


«  * 


read 

set  confirm  off 

« - 

♦  Data  Calculation  Section: 


locate  for  CAL_DATE  =  strtdate 

do  while  (.not.  eof())  .and.  (CAL_DATE  <=  enddate) 

ffl  9,10  say  "Generating  data  ..." 
store  CAL_DATE  to  mdate 
store  T0T_NUM  to  my_tot 
store  0  to  p_offset 
store  day_val  to  mtestdur 
max_dur  =  (mtestdur)  /  my_tot 
select  TIME 

for  loop_var  =  1  to  my_tot 
®  15,10  say  "Data  Point  #  " 

®  15,24  say  loop_v2tr 
append  blcink 

replace  CAL_DATE  with  mdate 
replace  TEST_DUR  with  mtestdur 
# - -  - - — -  - — - — - - 

♦  My  version  of  a  random  number  generator. 

♦  Tahes  the  system  time  and  finds  a  value  for 

♦  the  local  offset  of  failure  occurrence  within 

♦  a  time  "window"  by  using  sqrt()  and  modulo: 

num_sec  =  seconds () 
do  while  num_sec  >  max_dur 

num_sec  =  num_sec  sqrt(num_sec)  At  */,  is  the  modulus  operator 
enddo 


♦  Estimate  time  of  failure  from  number  of  partitions,  duration 

♦  of  partitions,  and  time  offset: 

failtime  =  (p_offsetrmax_dur)  +  (num_sec) 

replace  L_TIME_0CCUR  with  failtime  At  Local  Time  of  Occurrence 

replace  T_TIME_0CCUR  with  failtime  +  d.offset  At  Total  Time  of  Occurrence 


m_e  =  m_e  +  1 
replace  TOTAL  with  m_e 

p_offset  =  p_offset  +1 
next 

d_offset  =  d_offset  +  day_val 
select  COUNT 
skip 
enddo 
close  all 


At  Total  Failures 
At  Move  to  next  partition 

At  Move  to  next  test  period 


Else,  database  does  not  exist: 
else 

®  23,10  say  "TIME. DBF  Does  Not  Exist." 
wait  "Hit  any  key  to  continue  ..." 
end  if 

* - 

return  At  to  srtime.prg  module. 
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D.7  Software  Reliabtltiy  TIME. DBF  B-lB  Module 

*tf******m********^****^^:Hf***^^iii:*****m*m******m********<>‘***********  ************ 


*  * 

*  Title  :  Software  Reliability  TIME. DBF  B-IB  Module  (SRTBl.PRG)  ♦ 

*  Version  ;  3.3  * 

*  Date  :  2  Oct  91  ♦ 

*  Author  :  Capt  Joseph  J.  Stanko  ♦ 

*  Security  :  Unclassified  ♦ 

*  Purpose  :  This  program:  ♦ 

*  Generates  TIME. DBF  data  from  the  summ2a'y  COUNT. DBF  database  ♦ 

*  and  the  specific  B-IB  flight  test  database  BIDATA.DBF.  * 

BIDATA.DBF. 


As  this  uses  a  specific  database,  it  also  allows  the  user  to 
print  the  BISDATA.DBF  (sorted  version  of  BIDATA.DBF). 


Theory  ;  This  is  a  one  pass  program  that  generates  failure  times  from 

*  actual  test  durations.  It  was  a  little  involved,  as  there  were  ♦ 

*  instances  of  COUNT. DBF  dates  that  had  no  corresponding  flight  * 

*  test  times,  as  well  as  BIDATA.DBF  dates  that  had  no  ♦ 

*  corresponding  failure  occurrences.  This  required  a  “running  ♦ 

*  summation"  of  either  failures  or  test  times  until  one  caught  up  * 

*  with  the  other.  Once  dates  for  both  failures  and  times  were  * 

*  the  Scune,  that  was  considered  the  date  of  failure  (for  this  * 

*  database  only)  with  the  time  of  failure  assigned  within  the  * 

*  total  test  time  for  that  date.  While  this  might  not  tcike  into  ♦ 

*  account  the  possibility  that  failures  were  discovered  on  * 

*  previous  flights,  it  does  preserve  the  relationship  of  test  • 

*  durations  amd  intervals  to  occurrence  of  failures  (in  a  ♦ 

*  relative  manner) .  ♦ 

*  ♦ 

♦  Database  :  This  program  uses  three  databases:  ♦ 

♦  * 

*  BIDATA.DBF  -  This  is  a  summary  database  of  B-IB  flight  test  * 

*  hours  and  dates:  * 

*  * 

♦  Name  Type  Length  Decimal  Description  * 

*  -  -  -  -  - « 

*  DATE  Date  Date  of  mission  flown  * 

♦  FLT.HRS  N  7  2  Duration  of  mission  (in  hours)* 

♦  FLT  C  6  Mission  identifier  ♦ 

*  * 

*  COUNT. DBF  -  This  is  an  intermediate  summeo-y  database  of  * 

*  dates  and  number  of  failures :  * 

*  ♦ 

*  Name  Type  Length  Decimal  Description  * 

*  -  -  -  -  - * 

*  CAL_DATE  Date  Date  of  occurrence  of  failure  * 

*  SEV_C0DE_1  N  4  #  of  Severity  Code  1  failures  * 

*  SEV_C0DE_2  N  4  #  of  Severity  Code  2  failures  * 

*  SEV_C0DE_3  N  4  #  of  Severity  Code  3  failures  ♦ 

*  SEV_C0DE_4  N  4  #  of  Severity  Code  4  failures  ♦ 

*  SEV_C0DE_5  N  4  #  of  Severity  Code  5  failures  ♦ 

*  NO_SEV_CODE  H  4  #  of  failures  not  coded  * 

*  TOT_NUM  N  4  Total  Number  for  this  date  * 

*  TOTAL  N  4  Overal  total  of  failures  * 

*  ♦ 

*  TIME. DBF  -  This  is  a  final  database  of  dates  and  estimated  * 

*  failure  times  and  test  durations:  ♦ 

*  ♦ 

♦  Name  Type  Length  Decimal  Description  * 

*  -  -  -  -  - « 

*  CAL_DATE  Date  Date  of  occurrence  of  failure  * 

*  L_TIHE_OCC  N  10  2  "Local"  time  of  failure  occur  * 

*  (wrt  to  start  of  that  test)  * 

*  TEST_DUR  N  10  2  Duration  of  test  for  that  day  * 

*  T_TIME_0CC  N  10  2  "Total"  time  of  failure  occur  * 

*  (wrt  to  all  total  test  time)  ♦ 

*  TOTAL  N  4  Total  failures  to  that  point  * 

*  ♦ 

*  Modules  :  N/A  * 

*  * 


******************************************************************************* 


*  First,  make  sure  the  data  is  sorted: 
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«  «  «  «  * 


cleeir  screen 


if  .not.  fileC’BlSDATA.DBF") 

C  3,10  say  "Sorting  the  B1  Data  ... 
use  BIDATA 

sort  on  DATE  to  tempi 
close  all 

rename  templ.dbf  to  BlSDATA.dbf 
endif 


♦  —  -  —  -  —  -  —  — — - -  -  -  - - - - - — —  —  —  — 

*  Then,  check  to  see  if  user  wants  the  data  printed: 

store  "N"  to  prvar 

8  5,10  say  "Would  you  like  to  print  sorted  B-IB  data  (Y/K)?"  ; 
get  prvar  picture  "8K  !"  valid(prv2ir$"YN") 

read 

if  upper(prvar)  =  "Y" 

use  BISDATA 

store  1  to  LOG 

store  "  "  to  dbnzuDe 

store  "s"  to  printvar 

8  9,10  say  "What  is  DB  name?"  get  dbname  picture  "!!!!'.!!!" 
read 

8  11,10  say  "Send  to  (S)creen,  (P)rinter,  or  (F)ile?"  ; 

get  printvar  picture  “8K  !“  valid(printvar$"SPF") 

read 

if  upper(printvar)  =  "P" 
set  device  to  print 
pagelength  =  56 
elseif  upper (printvar)  =  "F" 
set  printer  to  SRBIDATA.PRN 
set  device  to  print 
pagelength  =  4000 
else 
clear 

pagelength  =  20 
endif 


8  13,10  say  "Printing  results  ..." 
do  while  .not.  eof() 
if  LOG  =  1 

8  LOG, 20  say  "Database  for  " 

8  LOG, 33  say  dbname 
store  LQC+2  to  LOG 
8  LOG, 10  say  "Date" 

8  LOG, 21  say  "Fit  Hrs" 

8  LOG, 30  say  "Fit  Kum" 

8  LOG+1,10  say  " - 

store  LOG  +  2  to  LOG 
endif 

8  LOG, 10  say  DATE 
8  LOG, 21  say  FLT.HRS 
8  LOG, 30  say  FLIGHT 
store  LOG  +  1  to  LOG 

if  LOG  =  pagelength 
store  1  to  LOG 
if  upper (printvar)  =  "S" 

wait  "Hit  any  key  to  continue  . . . 
clear 
endif 
endif 

skip 

enddo 

if  upper (printvar)  =  "P" 
set  device  to  screen 
elseif  upper (printvar)  =  "F" 
set  device  to  screen 
set  printer  to 
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«  «  n  «  » 


else 

wait  "Hit  acny  key  to  continue  . . . 
endif 
close  all 

endif 


Now  generate  TIME. DBF  data: 


lezur  screen 


First,  check  to  see  if  the  TIME. DBF  exists: 
if  fileC'TIME.DBF") 

♦ - 

♦  Variable  Section: 

use  COUNT  alias  COUNT 
use  TIME  alias  TIME 

e  a!  ACt"  ^ 

use  BISDATA  alias  B1 
select  COUNT 

store  0  to  d_offset 

store  0  to  in_e 

store  0  to  max_fail 

store  0  to  mtestdur 

store  0  to  my_tot 

store  0  to  nuin_sec 

store  0  to  p_offset 

store  CAL_DATE  to  mbldate 
store  CAL.DATE  to  mdate 
store  CAL.DATE  to  strtdate 
go  bottom 

store  CAL.DATE  to  enddate 
go  top 

♦  — - — - - - — - -  - - 

♦  Data  Entry  Section: 

set  confirm  on 

4)  3,10  say  "Enter  Starting  Date  for  Data  :"  get  strtdate  picture  "99/99/99" 

fl  4,10  say  "Enter  Ending  Date  for  Data  :"  get  enddate  picture  "99/99/99" 

read 

set  confirm  off 

♦  - — - — - — - —  —  — —  — - -  -  - - —  — - —  —  — - -  -  - - —  —  — - 

*  Data  Calculation  Section: 

locate  for  CAL_DATE  >=  strtdate 
select  B1 

locate  for  DATE  >=  strtdate 
select  COUNT 

do  while  (.not.  eofO)  .and.  (CAL_DATE  <=  enddate) 

Q  7,10  say  "Generating  data  ..." 
store  CAL.DATE  to  mdate 
store  T0T_NUM  to  my_tot 

select  B1 

store  0  to  mtestdur 
store  0  to  max_fail 

♦ - 

*  The  order  of  these  next  conditionals  acts  like  a  filter  to 

*  synch  the  test  dates  and  failure  dates. 

*  - 

*  First,  check  to  see  if  there  is  a  fit  record  for  corresponding 

*  failure  date.  If  not,  then  add  up  total  failures  until  we 

*  get  to  or  pass  the  next  fit  record: 

if  (DATE  >  mdate) 

store  DATE  to  mbldate 
select  COUNT 

do  while  (CAL.DATE  <  mbldate)  .and.  (.not.  eof()) 
skip 

store  (T0T_NUM  +  my_tot)  to  my_tot 


tA  Database  of  failure  COUNT  data 

&£  Database  of  failure  TIME  data 

Aft  Database  of  test  time  2md  duration  data 

ftft  Day  offset  to  determine  total  test  time 

ftft  Total  number  of  failures 

ftft  Test  var  for  failures  with  no  test  times 

ftft  Local  value  for  test  duration 

ftft  Local  total  number  of  failures 

ftft  System  time  (sec)  for  random  failure  times 
ftft  Partition  offset  for  failure  times 
ftft  Local  date  for  fit  test  occurrence 
ftft  Local  date  for  failure  occurrence 
ftft  Starting  date  for  data  einalysis 

ftft  Ending  date  for  data  analysis 
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enddo 

store  CAL_DATE  to  mdate 
select  B1 
endif 

—  —  —  —  —  —  —  —  —  — —  —  —  —  —  —  ———  —  — —  — 

*  Then,  check  to  see  if  the  failure  date  is  past  the  fit  record. 

*  If  so,  add  up  flight  times  for  interval  offset  value  until 

*  we  get  to  or  pass  the  next  failure  date  record: 

do  while  (DATE  <  mdate)  .and.  (.not.  eof()) 
store  (FLT_HRS*60)+d_off set  to  d_offset 
skip 
enddo 

*  - 

♦  If  we  pass  the  failure  date  again,  use  the  previous  fit  record 

*  test  time  for  test  duration  (must  decrement  the  day  offset 

*  by  the  test  duration  so  it’s  not  used  twice): 

if  (DATE  >  mdate) 
skip  -1 

store  (d_offset  -  (FLT_HRS*60) )  to  d_offset 
store  (FLT_HRS*60)+mtestdur  to  mtestdur 
skip 
endif 

♦  - —  —  —  — - — —  —  —  —  —  — - 

*  If  we  did  not  pass  the  failure  date  again,  than  the  dates 

*  must  be  equal. 

*  Add  up  multiple  test  durations  for  the  Scune  day  to  madce  sure 

*  the  entire  same  day  test  duration  is  used  for  failure  times: 

do  while  (DATE  =  mdate)  .and.  (.not.  eof()) 
store  (FLT_HRS*60)+mtestdur  to  mtestdur 
skip 
enddo 

4(  — —  —————————————  —  ——————————— 

*  This  is  an  error  check  to  make  sure  there  are  test  times  for 

*  the  failures: 

max_fail  =  (mtestdur)  /  my_tot 
if  max.fail  =  0 

C  20,10  say  "*•**♦**  TEST  DURATION  =  0  ♦♦♦**♦*•■ 
endif 

*  - - - - 

*  Now  that  we’ve  made  it  this  far,  assign  the  failure  times  remdomly 

*  (assuming  a  normal  distribution  for  ease  of  calculation)  within 

*  the  test  duration: 

select  TIME 
store  0  to  p_offset 
for  loop_v2u:  =  1  to  my_tot 

#  15,10  say  "Making  Entry  " 

®  15,24  say  loop_var  picture  "99" 

®  15,27  say  "of  " 

®  15,31  say  my_tot 

append  blank 
if  mdate  ="  /  /  " 

replace  CAL.DATE  with  enddate 

replace  CAL.DATE  with  mdate 
endif 

replace  TEST_DUR  with  mtestdur 

♦  - 

*  My  random  number  generator: 

num_sec  =  seconds () 
do  while  (num_sec/60)  >  max_fail 
num_sec  =  num_sec  V,  sqrt(num_sec) 
enddo 

failtime  =  (p_off set»max_fail)  +  (num_sec/60) 

replace  L_TIME_0CCUR  with  failtime  tt  Local  Occurrence  Time 


Aft  We  exceeded  the  end  of  file 


Aft  V,  is  the  modulus  operator 
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replace  T_TIME_OCCUR  with  failtime  +  d_offset  tt  Total  Occurrence  Time 
m_e  =  m_e  +  1 

replace  TOTAL  with  m_e  At  Total  Failures 

p_offset  =  p.offset  +1 
next 

d_offset  =  d_offset  +  mtestdur 
select  COUNT 
skip 
enddo 

* - 

♦  If  TIME. DBF  does  not  exist; 

gIs  6 

Q  23,10  say  "TIME. DBF  Does  Not  Exist." 
wait  "Hit  any  key  to  continue  ..." 
endif 

*  - 

return  tt  to  srtime.prg  module. 

#  4^  #  3^  *  *  4^  *  *  4t  ^  4c  4:  ♦  ^  #  *  *  *  *  *  *  4c  *  ^  4t  *  4c  4c  4k  *  *  ^ 
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«*«#***««  »  ««« 


D.8  Software  Reliability  TIME. DBF  Estimate  Module 


* 

Title  :  Software  Reliability  TIME. DBF  Estimate  Module  (SRTEST.PRG)  ♦ 

Version  :  3.3  ♦ 

Date  ;  25  Oct  91  * 

Author  :  Capt  Joseph  J.  Stanko  *■ 

Security  :  Unclassified  ♦ 

Purpose  :  This  program;  * 

Generates  TIME. DBF  data  from  the  summciry  COURT. DBF  database  * 
and  estimated  test  times  for  monthly  periods.  Generates  * 

TIMEDTE.DBF  data  the  same  way  (used  to  determine  the  failure  ♦ 
intensity  at  end  of  DT*E/start  of  OTAE) .  ♦ 


Theory  :  This  is  a  one  pass  program  that  generates  failure  times  from 

♦  estimated  test  durations.  The  user  is  prompted  for  number  of  * 

♦  months,  then  the  program  iterates  for  each  month  asking  the  * 

♦  user  for  the  estimated  test  time  for  that  month.  The  program  * 

♦  then  accesses  the  COURT. DBF  database,  and  locates  the  records  * 

♦  for  failures  occurring  during  that  month.  The  estimated  test  * 

♦  time  is  divided  by  the  number  of  days  in  the  month,  and  the  ♦ 

♦  times  are  summed  up  to  each  date  of  failure  data  for  local  * 

♦  values  of  test  duration.  For  example,  if  there  were  30  hrs  ♦ 

♦  estimated  for  September,  that  would  be  (assuming  the  standard  * 

♦  normal  distribution  again)  an  average  of  1  hr  a  day  testing.  * 

♦  While  this  is  probably  not  that  accurate,  taking  the  COURT. DBF  * 

♦  data  of  failures  and  summing  up  to  the  failure  dates  (in  this  * 

♦  case,  they  could  be  09/11/89,  09/15/89,  and  09/22/89)  that  ♦ 

♦  would  give  us  3  test  durations  of  11  hours,  4  hours,  and  * 

♦  7  hours,  with  the  additional  8  hours  rounded  into  the  offset  ♦ 

♦  for  the  following  month's  first  test  duration.  ♦ 

♦  This  is  the  best  I  can  do  as  I  am  working  with  summary  data.  * 

♦  Praise  the  Lord  Jesus  Christ!  ♦ 

♦  * 

*  Database  ;  This  progreun  uses  two  databases:  « 

*  * 

♦  COURT. DBF  -  This  is  an  intermediate  summary  database  of  * 

♦  dates  and  number  of  failures:  * 

♦  • 

*  Rame  Type  Length  Decimal  Description  * 

♦  -  -  -  -  - 

♦  CAL_DATE  Date  Date  of  occurrence  of  failure  * 

♦  SEV_C0DE_1  R  4  #  of  Severity  Code  1  failures  * 

♦  SEV_C0DE_2  R  4  #  of  Severity  Code  2  failures  ♦ 

♦  SEV_C0DE_3  N  4  #  of  Severity  Code  3  failures  * 

♦  SEV_C0DE_4  R  4  #  of  Severity  Code  4  failures  * 

♦  SEV_C0DE_5  R  4  #  of  Severity  Code  5  failures  * 

♦  N0_SEV_C0DE  R  4  #  of  failures  not  coded  * 

♦  T0T_RUM  R  4  Total  Riimber  for  this  date  * 

♦  TOTAL  R  4  Overal  total  of  failures  * 

♦  * 

*  TIMEDTE.DBF  -  This  is  DTAE  database  of  failure  time.  Used  * 

*  as  basis  for  OTAE  failure  intensity.  Same  * 

*  structure  as  TIME. DBF.  ♦ 

*  * 

*  TIME. DBF  -  This  is  a  final  database  of  dates  and  estimated  * 

*  failure  times  and  test  durations:  * 

*  * 

*  Rame  Type  Length  Decimal  Description  * 

*  -  -  -  -  - * 

*  CAL_DATE  Date  Date  of  occurrence  of  failure  * 

*  L_TIME_0CC  R  10  2  "Local"  time  of  failure  occur  • 

♦  (wrt  to  staort  of  that  test)  * 

•  TEST_DUR  R  10  2  Duration  of  test  for  that  day  • 

♦  T_TIME_0CC  R  10  2  "Total"  time  of  failure  occur  * 

*  (wrt  to  all  total  test  time)  * 

•  TOTAL  R  4  Total  failures  to  that  point  * 

•  ♦ 

•  Modules  ;  R/A  * 

*  ♦ 


**************************  ***«*«*«***««*«#*****»4i*«4i*4i**««*4t**«4t**««**»4i^^«*»4,« 
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*  * 


*  Determine  the  TIME. DBF  needed: 
store  "0"  to  dbvar 

C  23,20  say  ••(D)TftE  or  (0)T*E  Database?"  ; 

get  dbvar  picture  "•K  !"  valid(dbvar$"D0") 

read 

«  23,20  say  " 

*  - 

*  First,  check  to  see  if  either  TIME. DBF  or  TIMEDTE.DBF  exists: 

if  (fileC'TIME.DBF")  .and.  upper(dbvar)="0")  .or.  ; 
(fileC'TIMEDTE.DBF")  .and.  upper(dbvar)="D") 

« - 

*  Vauriable  Section: 


use  COUNT  alias  COUNT  tk 

select  2 

if  upper (dbvar )="0" 
use  TIME  alias  TIME  kk 

gIsg 

use  TIMEDTE  alias  TIME  kk 

endif 

select  COUNT 


store 

0 

to 

avg.time 

AA 

store 

0 

to 

d_off set 

AA 

store 

0 

to 

day.val 

AA 

store 

60 

to 

hour 

AA 

store 

0 

to 

last.month 

AA 

store 

0 

to 

m_e 

AA 

store 

0 

to 

m_off set 

AA 

store 

0 

to 

max.dur 

AA 

store 

"A" 

to 

mode.var 

AA 

store 

0 

to 

month.end 

AA 

store 

0 

to 

mtestdur 

AA 

store 

0 

to 

my.tot 

AA 

store 

0 

to 

num.days 

AA 

store 

0 

to 

num.month 

AA 

store 

0 

to 

num.sec 

AA 

store 

0 

to 

p.off set 

AA 

store  CAL.DATE  to  mdate  kk 

store  CAL.DATE  to  strtdate  AA 
go  bottom 

store  CAL.DATE  to  enddate  kk 
go  top 

« - 

*  Data  Entry  Section: 


Database  of  failure  COUNT  data 


Database  of  OTAE  failure  TIME  data 
upper (dbvar ) ="D" 

Database  of  DTAE  failure  TIME  data 


Average  test  time  per  month  for  OTAE  (hrs) 

Day  offset  to  determine  total  test  time 

Day  value  for  test  duration  (minutes) 

Number  of  minutes  in  an  hour 

Carry  over  time  from  previous  test  month 

Total  number  of  failures 

Month  offset  to  determine  total  test  time 

Max  partition  for  assigning  failure  times 

Mode  for  diagnostic  nrite  output 

Last  day  of  month  (varies  from  28  to  31) 

Local  value  for  test  duration  (min) 

Local  total  number  of  failures 
Number  of  days  used  to  calculate  mtestdur 
Total  number  of  months  for  OTAE  test 
Number  of  seconds  from  system  clock 
Partition  offset  for  local  failure  times 
Local  date  for  failure  occurrence 
Starting  date  for  data  eoialysis 

Ending  date  for  data  analysis 


clear  screen 
set  confirm  on 

#3,10  say  "Enter  Number  of  Months  for  Test:"  get  num.month  picture  "99" 

#4,10  say  "Enter  Starting  Date  for  Data  :"  get  strtdate  picture  "99/99/99" 

#6,10  say  "Enter  Ending  Date  for  Data  :"  get  enddate  picture  "99/99/99" 

read 

#6,10  say  "(A)uto  or  (S) ingle  Step  Mode?"  get  mode.var  picture 
read 

set  confirm  off 


*  Data  Calculation  Section: 

locate  for  CAL.DATE  =  strtdate  AA  Go  to  the  first  applicable  record 

store  CAL.DATE  to  mdate  AA  Update  mdate  to  match  strtdate 

for  loop.var  =  1  to  n\un_month  AA  I  assume  the  first  month  is  in  COUNT. DBF 

#  10,10  say  " 

#7,10  say  "Working  on  Test  Month  #" 

#7,33  say  loop_var 

if  loop.var  <  num.month  AA  Get  appropriate  input 

set  confirm  on 

#  8,10  say  "Enter  Avg  Test  Time/Month  (hrs):"; 
get  avg.time  picture  "9999.99" 

read 
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tt  Put  this  inside  both  or  get  2  in  fields 


* 


set  confirm  off 
C  8,8  clear  to  8,78 
else 

clear  gets  ftt  Remove  get  from  above 

set  confirm  on 

®  8,10  say  "Enter  Final  Month’s  Test  Time  (hrs):"; 
get  avg_time  picture  "9999.99" 

read 

set  confirm  off 
endif 

Q  10,10  say  "Generating  data  ..." 

♦ - 

*  Check  to  see  if  the  next  month  in  COUNT. DBF  is  a  consecutive 

*  month  of  testing,  including  wrap-ciround  (0  is  reset  condition): 

m_offset  =  month(CAL_DATE)-month(mdate) 
if  (m_offset=0)  .or.  (m_offset=l)  .or.  ; 

( (month(CAL_DATE)=l)  .emd.  (month(radate)=12)) 

♦ - - 

*  Determine  the  average  daily  test  time  euid  perform  all 

*  calculations  for  the  TIME. DBF  database: 


do  case 

case  (month(CAL_DATE)=4)  .or. 
.or.  (month(CAL_DATE)=9)  .or. 

month_end  =  30 
case  (month(CAL  DATE)=2) 
if  (year(CAL_DATE)*/.4)=0 
month_end  =  29 

qX  S  6 

month.end  =  28 


tit  Assign  daily  average  test  time 
(month(CAL_DATE)=6)  ; 
(month(CAL_DATE)=ll) 

tt  April,  June,  September,  cind  November 

tt  Check  for  Leap  Year 
tt  February  has  29  days 

tt  February  has  28  days 


endif 

otherwise 

month_end  =  31  A4  All  the  others  have  31  days 

endcase 

day.val  =  (avg_time*hour)/month_end 


(0  16,10  say  "Daily  Test  Time  =  "  tt  Echo  the  information 
fl  16,28  say  day_val 


♦ - 

*  For  each  entry  in  COUNT. DBF  within  the  same  month: 


store  day(CAL_DATE)  to  num.days  tt  Number  days  used  for  mtestdur 
store  CAL_DATE  to  mdate  tt  Reset  date  for  failure  occurrence 

store  0  to  prev_days  tt  Initialize  each  month 


do  while  (month(CAL_DATE)  =  month(mdate) ) ; 
•  suid.  (.not.  eofO) 


if  ((day(CAL_DATE)-num_days)<>0)  tt  Determine  #  of  days  of  test 
num_days  =  day(CAL_DATE)-prev_days 
endif 

store  day(CAL_DATE)  to  prev_days 


mtestdur  =  (day_val  ♦  num_days)  *4+  last_month 

store  CAL.DATE  to  mdate  44  Update  for  change  in  day 

store  T0T_NUM  to  my_tot 

max_dur  =  mtestdur  /  my.tot  44  Set  maximum  partition  time  duration 


0  17,10  say  "Num  Days  =  "  44 

6  17,21  say  num_days  44 

«  18,10  say  "Prev  Days=  " 

9  18,21  say  prev_days 

«  19,10  say  "MTESTDUR  =  " 

0  19,21  say  mtestdur 

C  20,10  say  "Max  Dur  =  " 

C  20,21  say  max.dur 

C  21,10  say  "  " 

if  mode_var  =  "S" 

wait  "Hit  any  key  to  continue 

endif 


Output  the  calculation  data 
to  verify  that  it  works 


* 
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*  Now  that  we've  got  the  test  duration  and  number  of  failures, 

^  assign  times  (assuming  normal  distribution  for  ease  of  calculation) 

*  within  the  test  duration: 

select  TIME 
store  0  to  p_offset 
for  loop2_var  =  1  to  my_tot 
®  15,10  say  "M2dting  Entry  " 

®  15,24  say  loop2_var  picture  "OO" 

®  15,27  say  "of  " 

®  15,31  say  my_tot 
append  blank 

replace  CAL.DATE  with  mdate 
replace  TEST_DUR  with  mtestdur 

* - 

*  My  random  number  generator: 

num_sec  =  seconds () 

do  while  (niuii_sec/60)  >  max_dur 

num_sec  =  num.sec  %  sqrt(num_sec)  Aft  %  is  the  modulus  operator 
enddo 

if  nvun_sec  =0  Aft  Check  for  0  time  interval 

num_sec  =  60  AA  and  set  to  a  min  value 

endif 

failtime  =  (p_offset*max_dur)  +  (num_sec/60) 

replace  L_TIME_0CCUR  with  failtime  AA  Local  Occurrence  Time 
replace  T_TIME_0CCUR  with  failtime+d_off set  AA  Total  Occurrence  Time 
m_e  =  m_e  +  1 

replace  TOTAL  with  m_e  AA  Total  Failures 

p_offset  =  p.offset  +1 
next 

d.offset  =  d.offset  +  mtestdur 

select  COUNT 

skip 

enddo 

- - - -  -  - - - - 

*  Check  for  time  at  end  of  month  after  last  COUNT  entry: 

if  .not.  eofO 
skip  -1 

if  (day (CAL_DATE)  <  month_end) 

num_days  =  month_end  -  day (CAL_DATE) 
d_offset  =  (num_days*day_val)  +  d_offset 
endif 
skip 
endif 

♦ - 

*  If  the  months  are  not  consecutive,  then  include  the  "between-test" 

*  time  as  part  of  the  offset: 

else 

if  (month(CAL_DATE)>month(mdate) )  AA  ie,  11  >  9 

m_of  f  s  et =month ( CAL_D ATE) -month (mdate) - 1 
else  AA  ie,  2  />  12 

m_of f set= ( 12-month (mdate) ) + (month(CAL_DATE)-l ) 
endif 

d_offset  =  (m_off set*avg_time*hour)  +  d_offset 
loop_V2ur  =  loop_var+m_off set-1 

store  CAL.DATE  to  mdate  AA  Reset  for  new  month’s  data 

endif 

®  21,10  say  "D.Offset  =  "  AA  Echo  the  information 
®  21,21  say  d.offset 
if  mode_veu:  =  "S" 

wait  "Hit  any  key  to  continue  — " 
endif 
next 
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close  all  tt  Saves  off  the  database  data 

* - 

♦  Else,  neither  TIME. DBF  nor  TIHEDTE.DBF  does  not  exist: 

else 

Q  23,20  say  "TIME  Database  Does  Not  Exist.  " 

wait  "Hit  any  key  to  continue  ..." 
endif 

♦  - 

return  tft  to  srtime.prg  module. 

4e  #  ^ 4c  14c  4c  ^  «  4c  4c %  :|t  4^  4^  4t  * 4t  #  # # 4c 4c  4t «  4c 4c 4c *  4c  4c 4c # «  4c 
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D.9  Software  Reliabiltiy  Execution  Time  Module 


* 

Software  Reliability  Execution  Time  Module  (SREXEC.PRG)  * 
3.3  * 
25  Oct  91  ♦ 
Capt  Joseph  J.  Steinko  * 


* 

♦  Title 

♦  Version 

♦ 

« 


Date 
Author 

*  Security 

*  Purpose 

* 

* 

* 

* 

* 

* 


Theory 


Database 


Unclassified 
This  progrcun: 

1. )  Performs  calculations  on  data  to  determine  initial 

paxcimeters  for  the  fitted  model. 

2. )  Performs  calculations  on  data  to  determine  goodness-of-f it 

for  the  Execution  Time  model. 

In  order  to  apply  the  Execution  Time  model  to  the  failure  data, 
initial  pareimeter  estimation  must  be  accomplished  from  the 
overall  data.  Once  the  parameters  axe  calculated,  they  are 
then  used  in  the  model  to  calculate  estimated  values  (such  as 
current  number  of  failures  and  failure  intensity)  along  with  the* 
9551  percent  confidence  intervals.  These  estimations  are  then 
compared  against  the  actual  data  to  determine  the 
goodness-of-f it . 

This  program  uses  one  database: 

TIME. DBF 


This  is  a  final  database  of  dates  eind  estimated 
failure  times  and  test  durations: 


Name 

Type 

Length 

Decimal 

Description 

* 

♦ 

CAL  DATE 

Date 

Date  of  occurrence  of  failure 

* 

L_TIME_0CC 

N 

10 

2 

"Local"  time  of  failure  occur 

* 

♦ 

(wrt  to  start  of  that  test) 

* 

♦ 

TEST.DUR 

N 

10 

2 

Duration  of  test  for  that  day 

* 

4t 

T.TIHE.OCC 

N 

10 

2 

"Total"  time  of  failure  occur 

* 

« 

(wrt  to  all  total  test  time) 

* 

* 

TOTAL 

N 

4 

Total  failures  to  that  point 

♦ 

*  Modules 

:  Calls  internal  procedures  BHATOTE 

and  BHATDTE. 

* 

***********************************  ******4>«****«*********«*«4t#  ***41  ************* 


Store  "0"  to  dbvar 

ffl  23,20  say  "(D)TftE  or  (O)TftE  Database?”  ; 

get  dbvar  picture  "8K  !”  valid(dbvar$"D0") 

read 

8  23,20  clear 

if  upper(dbvar)  =  "0" 
use  TIME  alias  TIME 
gXsg 

use  TIMEDTE  alias  TIME 
endif 


4c ............. — - - —  —  —  —  — - —  —  —  —  —  —  —  — - 

*  Variable  Section  (Note:  mu's  and  leunbda's  are  defined  later  on): 


go  bottom 

store 

M  fl 

to 

dbname 

ftft 

store 

0.0000001 

to 

delta 

kt 

store 

"M" 

to 

intervar 

kk 

store 

0.000 

to 

lambda.O 

kk 

store 

TOTAL 

to 

m_e 

kk 

store 

30 

to 

max_iter 

kk 

store 

1 

to 

num_iter 

kk 

store 

"S" 

to 

printvar 

kk 

store 

T_TIME_0CCUR  to 

t_e 

kk 

* 

Data  Entry 

Section : 

8  0,0 

clear 

set  confirm  on 

8  3,10  say  "Enter  Total  Test  Time: 
8  4,10  say  "Enter  Max  #  Iteration: 


Name  of  database  for  output  headers 

Accuracy  difference  for  parameters 

Data  output  intervals  (monthly  or  daily) 

Initial  failure  intensity  value 

Total  number  of  failures 

Max  iterations  to  perform  Newton-Raphson 

Current  iteration  number  for  Newton-Raphson 

Default  print  option  (screen) 

Total  test  time 


"  get  t_e  picture  "9999999.99" 

"  get  max_iter  picture  "999" 


n-2t) 


fl  5,10  say  "Enter  MLE  Delta  get  delta  picture  "9.99999999" 

if  upper(dbvar)="0" 

6  6,10  say  "Enter  Initial  Failure  " 

9  7,10  say  "Intensity  (0  for  none):"  get  lcunbda_0  picture  "9.999999999" 
endif 
read 

®  9,10  say  "What  is  DB  Neune  for  Output?"  get  dbneune  picture  "!!!!!!!!" 
read 

set  confirm  off 

Q  11,10  say  "Data  Output  Interval:  (ll)onthly  or  (D)aily:"  ; 
get  intervar  picture  "flK  !"  valid (intervar$"MD") 

read 

fi  13,10  say  "Send  Data  to  (S)creen,  (P)rinter,  or  (F)ile:"  ; 
get  printvar  picture  "®K  !"  valid(printvaur$"SPF") 

read 

^ - 

*  Data  Direction  Section: 

if  upper(printvar)  =  "F" 
if  upper(dbvar)  =  "0" 

®  15,10  say  "Sending  Data  to  File  SREXEC.PRN  ..." 
set  printer  to  SREXEC.PRH 
else  upper(dbvar)  =  "D" 

®  15,10  say  "Sending  Data  to  File  SREXECD.PRN  ..." 
set  printer  to  SREXECD.PRN 
endif 

set  device  to  print 
pagelength  =  4000 
elseif  upper (printvar)  =  "P" 

®  15,10  say  "Printing  Data  ..." 
set  device  to  print 
pagelength  =  55 
else 

®  0,0  clear 
pagelength  =  20 
endif 

♦  -  —  -  —  - - - - - - - - - - ....... - .... - ... - 

*  Data  Calculation  Section:  Maximum  Likelihood  Estimation 

®  3,7  say  "MLE  Calculations  for" 

®  3,31  say  dbname 

®  3,40  say  "using  Musa  Execution  Time  Model:" 

♦  — - .... - ..... - ... - 

*  Med(e  initial  model  parameter  estimation,  and  sum  failure  occurance 

*  times  to  make  calculations  easier: 

b_hat  =  l/(t  e) 
go  top 

®  5,10  say  "Total  Failures:  m_e  =  " 

®  5,43  say  m_e  picture  "99999.99" 

®  6,10  say  "Failure  Data  End  Time:  t_e  =  " 

®  6,43  say  t_e  picture  "99999999.99" 

®  7,10  say  "Initial  Model  Param  Est:  b_hat  =  " 

®  7,43  say  b_hat  picture  "99.999999999" 

« - 

*  Determine  a  better  estimation  for  b_hat  by  making  f_stat  as  close 

*  as  possible  to  0  (uses  Newton-Raphson  method) : 

if  upper(printvar)  =  "P" 
set  device  to  screen 

®  17,10  say  "Refining  b_hat  Estimate,  Please  Wait  ..." 
set  device  to  print 
elseif  upper (printvar)  =  "S" 

®  8,10  say  "Refining  b_hat  Estimate,  Please  Wait  ..." 
endif 

« - 

*  Iterate  while  out  of  tolereoice  or  within  allotted  looping  time: 
num_iter  =  1 
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not_in_tol  =  .T. 

do  while  (not_in_tol)  .eind.  (nuin_iter  <=  max_iter) 


♦  Determine  the  f(b_hat)  aind  f’(b_hat)  for  Newton-Raphson  method 

*  based  on  known  initial  value  of  failure  intensity  (lambda_0) : 

if  upper (dbvar)  =  "0"  .and.  ;  tk  Different  equation  set  for  OTtE  using 
launbda_0  <>  0.0  ftft  previous  DTftE  failure  intensity  # 

f_stat  =  ((m_e’t‘b_hat)/(l-exp(-b_hat*t_e)))  -  lcunbda_0 

fp_stat  =  (((l-exp(-b_hat*t_e) )*m_e)-((m_e*b_hat)*(t_e*exp(-b_hat*t_e) ))) 
/  ((l-exp(-b_hat*t_e) )“2) 


* 


♦  Determine  the  f(b_hat) 

*  with  no  clues  at  all: 

and 

f’(b_hat)  for 

Newton-Raphson  method 

else 

kk 

kk 

We’re  either  looking  at  DTftE  data 
or  OTftE  data  without  a  priori  info 

go  top 

t_i  =  0 
do  while 

.not.  eof() 

kk 

Summation  of 

failure  occur  times 

t_i  =  t_i  +  T_TIME_0CCUR 
skip 
enddo 


f_stat  =  (m_e/b_hat)  -  ((m_e*t_e)/(exp(b_hat*t_e)-l))  -  t_i 

fp_stat  =  (m_e  *  (-l/b_hat*2) )  -  ; 

(m_e*t_e)*((-l*t_e^exp(b_hat*t_e))/(exp(b_hat*t_e)-l)'2) 

end  if 


—  —  —  —  —  —  —  —  —  —  —  —  —  —  — 

•  The  rest  is  the  same  for  both  cases  from  above: 


bp_hat  =  b_hat  -  (f_stat/fp_stat) 

kk 

Burden  and  Faires  Step  #3 

if  abs(bp_hat-b_hat)  <  delta 
not_in_tol  =  .F. 
endif 

kk 

Check  for  within  tolerance  delta 

if  upper(printvar)  =  "P" 

kk 

Output  the  data  as  it  is  calculated 

1 

set  device  to  screen 

kk 

to  verify  convergence 

®  19,0  clear 
®  19,10  say  "b_hat  =  " 

ffl  19,20  say  b_hat 

®  20,10  say  "bp_hat  =  " 

®  20,20  say  bp_hat 

®  21,10  say  "F.Stat  =  " 

®  21,20  say  f_stat 

®  22,10  say  "Fp.Stat  =  " 

®  22,20  say  fp_stat 

®  17,10  say  "Refining  b_hat  Estimate,  Please  Wait  ..." 
set  device  to  print 

elseif  upper(printvar)  =  "S"  &ft  Same  thing  here,  but  must  use 

®  9,0  clear  tk  different  screen  output  positions. 

®  9,10  say  "b_hat  =  " 

®  9,20  say  b_hat 

Q  10,10  say  "bp.hat  =  " 

®  10,20  say  bp_hat 

«  11,10  say  "F_Stat  =  " 

®  11,20  say  f_stat 

A  12,10  say  "Fp.Stat  =  " 

0  12,20  say  fp_stat 

®  8,10  say  "Refining  b_hat  Estimate,  Please  Wait  ..." 
endif 

b_hat  =  bp_hat 
num_iter  =  num_iter  +  1 

enddo 
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- — - - - — - - - -  - - — - 

♦  Output  additional  data  on  refined  values: 

if  upper(printvar)  =  "P" 
set  device  to  screen 
®  19,0  clear 

®  19,10  say  "Printing  Data  ..." 
set  device  to  print 
elseif  upper(printvar)  =  "S" 

®  9,0  clear 
end  if 

if  num.iter  >  max_iter 

®  9,10  say  "Method  Failed  After  " 

®  9,30  say  niax_iter  picture  "999" 

®  9,34  say  "Iterations." 
end  if 

®  10,10  say  "Final  Model  Param  Est:  b_hat  =  " 

®  10,43  say  b_hat  picture  "99.999999999" 

®  11,10  say  "  " 

4c - -  -  -  - - - - - - -  - - - - - - - ....... - - - 

♦  Data  Calculation  Section:  Parameters  and  Confidence  Intervals 

♦  Determine  the  Expected  (Fisher)  Information,  cind  then 

♦  Calculate  95*/.  Confidence  Intervals: 

®  12,10  say  "Parcimeter  Calculations:  nu_0,  lajnbda_0,  2md  95th  Percentile:" 

fisher  =  m_e  *  (  (l/b_haf2)  -  (  (t_e"2*exp(b_hat*t_e) )  ; 

/  (exp(b_hat*t_e)-l)“2  )) 

Aft  Initial  model  parameter 

b_hat_low  =  b_hat  -  (1 .96/sqrt(f isher))  ftft  From  Z  statistic 

b_hat_hi  =  b_hat  +  (1 .96/sqrt(f isher)) 

ftft  Derived  model  parameter 

b_0  =  (m_e)  /  (l-exp(-(b_hat*t_e)))  ftft  Calculated  from  parameter 

b_0_low  =  (m_e)  /  (l-exp(-(b_hat_low*t_e)))  ftft  b_hat. 

b_0_hi  =  (m_e)  /  (l-exp(-(b_hat_hi*t_e))) 

ftft  Total  Failures  at  t=infinity 
nu_0  =  b_0  ftft  By  Definition 

nu_0_low  =  b_0_low 

nu_0_hi  =  b_0_hi 

ftft  Initial  Failure  Intensity 

if  lambda.O  =0  ftft  If  we  don't  have  a  user 

lambda_0  =  b_0  ♦  b_hat  ftft  input,  calculate  it 

end  if 

lambda_0_low  =  b_0_low  ♦  b_hat_low  ftft  Varying  b_hat  effects  both 

lambda_0_hi  =  b_0_hi  ♦  b_hat_hi  ftft  b_hat  and  b_0,  etc. 

®  14,10  say  "Expected  (or  Fisher)  Value  =  " 

®  14,48  say  fisher  picture  "9999999999.999999999" 

®  16,10  say  "95*/,  Boundary:  nu_0_low  =  " 

®  16,48  say  nu_0_low  picture  "99999999.99" 

®  17,10  say  "Total  Estimated  Failures:  nu_0  =  " 

®  17,48  say  nu_0  picture  "99999999.99" 

®  18,10  say  "95*/,  Boundary:  nu_0_hi  =  " 

®  18,48  say  nu_0_hi  picture  "99999999.99" 

®  20,10  say  "95*/.  Boundeory:  lcunbda_0_low  =  " 

®  20,48  say  launbda_0_low  picture  "99.999999999" 

®  21,10  say  "Initial  Failure  Intensity:  lambda_0  =  " 

®  21,48  say  lambda_0  picture  "99.999999999" 

®  22,10  say  "95*/,  Boundary:  lambda_0_hi  =  " 

®  22,48  say  lambda_0_hi  picture  "99.999999999" 

®  23,10  say  "  " 

- 

♦  Output  of  Model  Results: 

if  upper(printvar)  =  "P" 
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set  print  on 
eject 

set  print  off 

elseif  upper(printvar)  =  "S" 

wait  "Hit  any  key  to  continue _ ** 

cleeir 

endif 

go  top 
LOC  =  1 

do  while  .not.  eof() 

if  LOC  =1  tt  Header  information 

®  LOC.IO  say  "Generating  Plot  Data  for" 
fi  LOC, 35  say  dbname 

a  LOC+1.5  say  " - " 

a  LOC+1.51  say  " - " 

LOC  =  LOC  +  3 
endif 

store  CAL_DATE  to  mdate 
store  T_TIME_OCCUR  to  tau 
store  TOTAL  to  mu 


- - - - - — - - 

*  Calculate  this  info  for  each  pass  in  the  loop: 

♦  Failures  Experienced  at  time  t=tau 

mu_tau  =  nu_0  *  (1  -  exp(-(lambda_0/nu_0)*tau) ) 
mu_tau_low  =  nu_0_low  ♦  (1  -  exp(-(lambda_0/nu_0_low)*tau) ) 
mu_tau_hi  =  nu_0_hi  *  (1  -  exp(-(lambda_0/nu_0_hi)*tau) ) 

*  Failure  Intensity  at  time  t=tau 
lambda_tau  =  lambda_0  *  exp(-(lambda_0/nu_0)*tau) 
lambda_t_low  =  lambda_0_low  ♦  exp(-(lambda_0_low/nu_0_low)*tau) 
lambda_t_hi  =  lambda_0_hi  *  exp(-(lambda_O_hi/nu_0_hi)*tau) 

♦  Failure  Intensity  at  mu  failures  experienced 
lambda_mu  =  lambda_0  *  (l-(mu/nu_0)) 
lambda_m_low  =  lambda_0_low  *  (l-(mu/nu_0_low)) 
lambda_m_hi  =  lambda_0_hi  ♦  ( l-(mu/nu_0_hi)) 


♦  —  —  —  —  ~  — —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  — 

*  Now  output  the  info  for  each  pass  in  the  loop: 


a  LOC, 10  say 
a  LOC, 16  say 
a  LOC, 30  say 
a  LOC, 44  say 
L0C=L0C+1 
a  LOC, 10  say 
a  LOC , 16  say 
a  LOC, 30  say 
a  LOC, 44  say 
L0C=L0C+1 
a  LOC, 10  say 
a  LOC, 16  say 
a  LOC, 30  say 
a  LOC, 44  say 
L0C=L0C+1 
a  LOC, 10  say 
a  LOC, 28  say 
a  LOC, 43  say 
a  LOC, 60  say 
L0C=L0C+1 
a  LOC , 10  say 
a  LOC, 28  say 
a  LOC, 43  say 
a  LOC, 60  say 
L0C=L0C+1 
a  LOC, 10  say 
a  LOC, 28  say 
a  LOC, 43  say 


"Day  :  " 
mdate 

"mu(tau)  low  =  " 
mu_tau_low  picture  "9999.99" 

"mu  =  " 

mu  picture  "9999.99" 

"mu(tau)  =  " 
mu_tau  picture  "9999.99" 

"tau  =  " 

tau  picture  "9999999.99" 

"mu (tau)  hi  =  " 
mu_tau_hi  picture  "9999.99" 

"lambda(tau)  low  =  " 
leimbda_t_low  picture  "99.999999999" 
"lambda(mu)  low  =  " 
lambda_m_low  picture  "99.999999999" 

"lambda(tau)  =  " 
lambda_tau  picture  "99.999999999" 
"lambda(mu)  =  " 
larobda_mu  picture  "99.999999999" 

"lambda(tau)  hi  =  " 
l2unbda_t_hi  picture  "99.999999999" 
"lambda(mu)  hi  =  " 
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<D  L0C,60  say  lEunbda_m_hi  picture  "99.999999999" 

L0C=L0C+2 

ii  LOC  >  pagelength 
LOC  =  1 

if  upper (printvar)  =  "S" 

wait  "Hit  any  key  to  continue  ..." 
clear 
endif 
endif 

♦  — - -  -  -  -  -  -  -  - - —  — 

*  Skip  for  either  every  entry  or  for  first  entry  of  each  month: 
skip 

if  upper(intervar)="M" 

do  while  (month(CAL_DATE)  =  month(mdate))  .and.  (.not.  eof()) 
skip 
enddo 
endif 

enddo 

* - 

if  upper(printvar)  =  "F" 
set  device  to  screen 
set  printer  to 

?  chr(7)  ftt  Wake  me  up  when  donef 

elseif  upper (printvar)  =  "F" 
set  device  to  screen 
set  print  on 
eject 

set  print  off 
else 

wait  "Hit  any  key  to  continue  ..." 
endif 

* - 

close  all 

return  tk  to  SRSAS 

i),  ,),*:*******************************************************************•******** 
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D.IO  Software  Reliability  Logarithmic  Poisson  Execution  Tune  Module 


♦  * 

*  Title  :  Software  Reliability  Logarithmic  Poisson  Module  (SRLOG.PRG)  * 

*  Version  ;  3.3  ♦ 

*  Date  :  15  Oct  91  * 

*  Author  :  Capt  Joseph  J .  Stanko  * 

*  Security  :  Unclassified  * 

*  Purpose  :  This  program:  * 

*  1.)  Performs  calculations  on  data  to  determine  initial  * 

*  parameters  for  the  fitted  model.  * 

*  2.)  Performs  calculations  on  data  to  determine  goodness -of -fit  * 

*  for  the  Logarithmic  Execution  Time  model.  * 

*  * 

*  Theory  :  In  order  to  apply  the  Logcirithmic  Execution  Time  model  to  the  * 

*  failure  data,  initial  parameter  estimation  must  be  * 

*  accomplished  from  the  overall  data.  Once  the  parameters  * 

*  cire  calculated,  they  are  then  used  in  the  model  to  calculate  * 

*  estimated  values  (such  as  current  number  of  failures  and  * 

*  failure  intensity)  along  with  the  95*/,  confidence  intervals.  * 

*  These  estimations  are  then  compared  against  the  actual  data  to  * 

*  determine  the  goodness-of-f it .  * 

*  * 

*  Database  :  This  progr^Lm  uses  one  database:  * 

♦  * 

*  TINE. DBF  -  This  is  a  final  database  of  dates  eoid  estimated  * 

*  failure  times  and  test  durations:  ♦ 

*  ♦ 

♦  Name  Type  Length  Decimal  Description  ♦ 

*  -  -  -  -  - ^ 

*  CAL_DATE  Date  Date  of  occurrence  of  failure  * 

*  L_TiME_OCC  N  10  2  "Local"  time  of  failure  occur  * 

*  (wrt  to  start  of  that  test)  * 

*  TEST_DUR  N  10  2  Duration  of  test  for  that  day  ♦ 

*  T_TIME_0CC  N  10  2  "Total"  time  of  failure  occur  * 

*  (wrt  to  all  total  test  time)  * 

*  TOTAL  N  4  Total  failures  to  that  point  ♦ 

*  ♦ 

•  Modules  :  H/A  * 

*  * 


*^*^i$i^*^*i¥m******^**^^^^tt^^^ttit^^^^0^m***’¥*m*****^m^***<i‘ ******  *********‘***  ******* 

store  "0"  to  dbvar 

<0  23,20  say  "(D)TftE  or  (0)T&E  Database?"  ; 

get  dbvar  picture  "®K  !"  valid(dbvaur$"D0") 

read 

®  23,20  clear 

if  upper(dbvar)  =  "0"  Aft  Use  0T*E  version  of  time  database 

use  TIME  alias  TINE 
ds  6 

use  TINEDTE  alias  TIME  ftft  Use  DTRE  version  of  time  database 

endif 


♦  Variable  Section  (Note:  mu’s  and  lambda’s  are  defined  later  on): 
go  bottom 

store  "  "  to  dbname  Naune  of  database  for  output  header 

store  0.0000001  to  delta  tt  Accuracy  difference  of  parameters 

store  "M"  to  intervar  Aft  Data  output  intervals  (monthly  or  daily) 

store  0.000  to  lambda.O  Initial  failure  intensity  value 

store  TOTAL  to  m_e  Aft  Total  number  of  failures 

store  30  to  max_iter  AA  Max  iterations  to  perform  Newton-Raphson 

store  "S"  to  printvar  AA  Default  print  option  (screen) 

store  T_TIME_0CCUR  to  t_e  AA  Total  test  time 

♦  - 

♦  Data  Entry  Section: 

®  0,0  clear 
set  confirm  on 

«  3,10  say  "Enter  Total  Test  Time:"  get  t_e  picture  "9999999.99" 

®  4,10  say  "Enter  Max  #  Iteration:"  get  max_iter  picture  "999" 

C  5,10  say  "Enter  MLE  Delta  :"  get  delta  picture  "9.99999999" 
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if  upper (dbvar)  =  "0" 

C  6,10  say  "Enter  Initial  Failure  " 

®  7,10  say  "Intensity  (0  for  none):"  get  lambda_0  picture  "9.999999999" 
endif 
read 

®  9,10  say  "What  is  DB  Name  for  Output?"  get  dbncune  picture  "M  M  !  !  M" 
read 

set  confirm  off 

®  11,10  say  "Data  Output  Interval:  (M)onthly  or  (D)aily:"  ; 
get  intervar  picture  "®K  !"  valid(intervar$"MD") 

read 

®  13,10  say  "Send  Data  to  (S)creen,  (P)rinter,  or  (F)ile:"  ; 
get  printvar  picture  "®K  !"  valid{printvar$"SPF") 

read 

- - 

♦  Data  Direction  Section: 

if  upper (printvar)  =  "F" 
if  upper (dbvar)  =  "0" 

®  15,10  say  "Sending  Data  to  File  SRLOG.PRN  ..." 
set  printer  to  SRLOG.PRN 
else  upper(dbvar)  =  "D" 

®  15,10  say  "Sending  Data  to  File  SRLOGD.PRN  ..." 
set  printer  to  SRLOGD.PRN 
endif 

set  device  to  print 
pagelength  =  4000 
elseif  upper (printvar)  =  "P" 

®  15,10  say  "Printing  Data  ..." 
set  device  to  print 
pagelength  =  55 
else 

®  0,0  clear 
pagelength  =  20 
endif 


- - 

*  Data  Calculation  Section:  Maximum  Likelihood  Estimation 

®  3,7  say  "HLE  Calculations  for" 

®  3,31  say  dbname 

®  3,40  say  "using  Logarithmic  Poisson  Model:" 

*  - 

*  Make  initial  model  parameter  estimation,  and  sum  failure  occurrence 

*  times  to  make  calcuations  easier: 

b_hat  =  l/(t_e)  Musa's  recommended  guess 

go  top 

®  5,10  say  "Total  Failures:  m_e  =  " 

®  5,43  say  m_e  picture  "99999.99" 

®  6,10  say  "Failure  Data  End  Time:  t_e  =  " 

®  6,43  say  t_e  picture  "99999999.99" 

®  7,10  say  "Initial  Model  Param  Est:  b_hat  =  " 

®  7,43  say  b_hat  picture  "99.999999999" 

*  - 

♦  Determine  a  better  estimation  for  b_hat  by  making  f_stat  as  close 

♦  as  possible  to  0  (uses  Newton-Raphson  method) : 

num_iter  =  1 
not_in_tol  =  .T. 

if  upper(printvar)  =  "P" 
set  device  to  screen 

®  17,10  say  "Refining  b_hat  Estimate,  Please  Wait  ..." 
set  device  to  print 
elseif  upper(printvar)  =  "S" 

®  8,10  say  "Refining  b_hat  Estimate,  Please  Wait  ..." 
endif 

♦  - 


D-33 


*  Iterate  while  out  of  tolerance  or  within  alloted  looping  time: 
do  while  (not_in_tol)  .and.  (num_iter  <=  max_iter) 


♦  —  -  -  -  -  -  -  -  -  -  - - 

♦  Determine  the  f(b_hat)  and  f'(b_hat)  for  Newton-Raphson  Method 

*  based  on  a  known  initial  value  of  failure  intensity  (lambda.O) : 


if  upper(dbvar)="0"  .euid.  ;  tic  Different  equation  set  for  OT*E  using 

lambda_0  <>  0.0  tt  previous  DT4E  failure  intensity  # 

b_one  =  (1  +  (b_hat*t_e))  tt  Shortens  equation  notation 

f_stat  =  ( (m_e*b_hat)  /  log(b_one))  -  lajnbda_0 

fp.stat  =  (((log(b_one))*m_e)-((m_e’»b_hat)*(t_e/b_one)))  /  ((log(b_one))*2) 
* - 

*  Determine  the  f(b_hat)  and  f’(b_hat)  for  Newton-Raphson  Method 

♦  with  no  initial  clues  at  all: 


else 


go  top 


Aft  Ue're  either  looking  at  DTftE  data 
ftft  or  OTkE  data  without  a  priori  info 

tb  Requires  looping  thru  database  again 


t_i_sum  =0  bb  Summation  of  failure  occur  times  (ti) 

t_i2_sum  =0  bb  Sum  of  squcire  of  fail  occur  times  (ti) 

do  while  .not.  eof() 

t_i_sum  =  t_i_sum  +  (1/  (1+  (b_hat*T_TIME  OCCUR))) 
t_i2_sum  =  t_i2_sum  +  (-T_TIME_0CCUR  /  (1  +  (b_hat*T_TIME_0CCUR))*2) 
skip 
enddo 

b_one  =  (1  +  (b.hat’ft.e))  bb  Shortens  equation  notation 

f_stat  =  (l/b_hat)*(t_i_sum)  -  ( (m_e*t_e)/{b_one  *  log(b_one))) 


fp.stat  =  ((l/b_hat)  *  t_i2_sum)  +  ((t_i_sum)  ♦  (-l/b_hat“2)) 
-  (  (-(m_e*t_e‘2)  *  (1+  log(b_one))) 

/  (b_one  ♦  log(b_one))*2  ) 


end  if 


- - — 

*  The  rest  of  this  is  the  same  for  either  case  from  above: 

bp_hat  =  b_hat  -  (f _stat/f p_stat)  bb  Burden  b  Faires  Step  #3 

if  abs(bp_hat-b_hat)<delta  bb  Check  for  within  toleremce  delta 

not_in_tol  =  .F. 
endif 


if  upper (printvar)  =  "P"  bb  Output  the  data  as  it  is  calculated 

set  device  to  screen  bb  to  verify  convergence. 

C  19,0  clear 
C  19,10  say  "b_hat 

0  19,20  say  b_hat 

0  20,10  say  "bp_hat  =" 

C  20,20  say  bp_hat 

e  21,10  say  "F_Stat  =  " 

fl  21,20  say  f_stat 

0  22,10  say  "Fp.Stat  =  " 

fi  22,20  say  fp_stat 

C  17,10  say  "Refining  b_hat  Estimate,  Please  Wait  ..." 
set  device  to  print 


elseif  upper(printvar) 
(D  9,0  cleea* 
fi  9,10  say  "b_hat 
fi  9,20  say  b_hat 
fi  10,10  say  "bp_hat 
fi  10,20  say  bp_hat 
fi  11,10  say  "F_Stat 
fi  11,20  say  f_stat 
fi  12,10  say  "Fp_Stat 
fi  12,20  say  fp.stat 
fi  8,10  say  "Refining 


=  "S"  bb  Same  thing  here,  but  must  use 

bb  different  screen  position  for  output. 


M 


b_hat  Estimate,  Please  Wait  ... 
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endif 


b_hat  =  abs(bp_hat) 
nuin_iter  =  nujn_iter  +  1 

enddo 


*  Output  additional  data  on  refined  values: 

if  upper(printvaur)  =  "P" 
set  device  to  screen 
C  19,0  clear 

•  19,10  say  "Printing  Data  ..." 
set  device  to  print 
elseif  upper(printvar)  =  "S" 

C  9,0  clear 
endif 

if  nuin_iter  >  max_iter 

a  9,10  say  "Method  Ended  After  " 
a  9,29  say  max_iter  picture  "999" 
a  9,33  say  "Iterations." 
endif 


a  10,10  say  "Final  Model  Param  Est;  b_hat  =  " 
a  10,43  say  b_hat  picture  "99.999999999" 
a  11,10  say  "  " 

- 

*  Data  Calculation  Section:  Parameters  and  Confidence  Intervals 

*  Determine  the  Expected  (Fisher)  information,  and  then 

*  Calculate  95*/.  Confidence  Intervals: 

a  12,10  say  "Parameter  Calculations:  theta,  lambda.O,  and  95th  Percentile:" 
b_one  =  (1  +  (b_hat*t_e)) 


fisher  =  m_e  *  (  ((2*t_e)/(b_hat*b_one*log(b_one))) 
-  (  (l/(2*b_hat*2*log(b_one))) 

♦  (1  -  (l/b_one"2))) 

“  (  (t_e'2*(log(b_one)+l)) 

/  ((b_one*log(b_one))*2)) 

) 


b_hat_low 

=  b.hat  -  (1 .96/sqrt(f isher)) 

fri 

From  Z  statistic. 

b_hat_hi 

=  b.hat  +  (1.96/sqrt (fisher)) 

b_0 

(m_e)  /  log(l+(b_hat*t_e)) 

ftft 

Calculated  from  pareuneter 

b_0_low  = 

(m_e)  /  log(l+(b_hat_low*t_e)) 

ftft 

b.hat . 

b_0_hi  = 

(m_e)  /  log(l+(b_hat_hi*t_e)) 

theta 

=  l/b_0 

tk 

By  definition. 

theta_loH 

=  1/b  0  low 

theta_hi 

=  l/b_0_hi 

lambda. 0 

=  b_0  ♦  b_hat 

kk 

Varying  b_hat  affects  both 

leuBbda.O.low  =  b_0_low  *  b_hat_low 

kk 

b.hat  and  b_0,  etc. 

lambda_0_hi  =  b_0_hi  *  b_hat_hi 


a 

14, 

,10 

say 

a 

H, 

,48 

say 

a 

16, 

,10 

say 

a 

16, 

,48 

say 

a 

17, 

,10 

say 

a 

17, 

,48 

say 

a 

18, 

,10 

say 

a 

18, 

,48 

say 

"Expected  (or  Fisher)  Value  = 

fisher  picture  "9999999999.999999999" 
"95"y(  Boundary:  theta_loB  = 

theta_low  picture  "99.999999999" 
"Failure  Intensity  Decay:  theta  = 

theta  picture  "99.999999999" 

"95*/,  Boundary:  theta_hi  = 

theta_hi  picture  "99.999999999" 


C  20,10  say 
C  20,48  say 
C  21,10  say 
a  21 ,48  say 
a  22,10  say 
a  22,48  say 


"957,  Boundary:  lambda.O.lov  = 

lambda_0_lov  picture  "99.999999999" 
"Initial  Failure  Intensity:  lambda.O  = 
lambda.O  picture  "99.999999999" 

"95*/,  Boundary:  lambda_0_hi  = 

lambda_0_hi  picture  "99.999999999" 
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M  <1 


•  23,10  say 


*  Output  of  Model  Results: 

if  upper(printvar)  =  "P" 
set  print  on 
eject 

set  print  off 

elseif  upper(printvar)  =  "S" 

wait  "Hit  any  key  to  continue  ... 
clear 
end  if 


go  top 
LOG  =  1 

do  while  .not.  eof() 


if  LOG  =  1 

a  LOG, 10  say  "Generating  Plot  Data  for" 
a  LOG, 35  say  dbname 

a  LOG+1,5  say  " - 

a  LOG+1,51  say  " - 

LOG  =  LOG  +  3 
endif 

store  GAL_DATE  to  mdate 
store  T_TIME_OGGUR  to  tau 
store  TOTAL  to  mu 


kt  Header  information: 


*  Galculate  this  info  for  each  pass  in  the  loop: 

mu_tau  =  (1/theta)  ♦  Iog((lambda_0*theta*tau)+1) 
mu_tau_hi  =  (l/theta_hi)  *  Iog((lambda_0*theta_hi*tau)+1) 
if  ( (Iambda_0*theta_low*tau)+1)  >  0 

mu_tau_low  =  (l/theta.low)  *  Iog((lambda_0*theta_low*tau)+1) 
brackets  =  .F. 
else 

mu_tau_low  =  abs(mu_tau_hi  -  mu.tau)  +  mu_tau 
brackets  =  .T. 
endif 

lambda_tau  =  lambda.O  /  ((lambda_0*theta*tau)+l) 
lambda_t_low  =  lambda_0_low  /  ( (Iambda_0_low*theta_low*tau)+1) 
lambda_t_hi  =  lambda_0_hi  /  ((Iambda_0_hi*theta_hi*tau)+1) 

lcunbda_mu  =  l2unbda_0  *  exp(-theta*mu) 
lambda_m_low  =  lambda_0_low  *  exp(-theta_low*mu) 
lambda_m_hi  =  lambda_0_hi  *  exp(-theta_hi*mu) 


- -  -  - - -  -  -  -  -  -  -  - 

*  Now  output  the  info  for  each  pass  in  the  loop: 


a  LOG, 10  say 
a  LOG, 16  say 
a  LOG, 30  say 
a  LOG, 44  say 
L0G=L0G+1 
a  LOG , 10  say 
a  LOG, 16  say 
a  LOG, 30  say 
a  LOG, 44  say 
L0G=L0G+1 
a  LOG, 10  say 
a  LOG, 16  say 
a  LOG, 30  say 
a  LOG. 44  say 
L0G=L0G+1 
a  LOG, 10  say 
a  LOG. 28  say 
a  LOG, 43  say 
if  brackets 


"Day  :  " 
mdate 

"mu(tau)  low  =  " 
mu_tau_iow  picture  "9999.99" 

"mu  =  " 

mu  picture  "9999.99" 

"mu(tau)  =  " 
fflu.tau  picture  "9999.99" 

"tau  =  " 

tau  picture  "9999999.99" 

"mu(tau)  hi  =  " 
mu_tau_hi  picture  "9999.99" 

"lafflbda(tau)  low  =  " 
lambda_t_low  picture  "99.999999999" 
"lambda(mu)  low  =  " 


a  LOG. 60  say  "(" 

a  LOG, 61  say  lambda_m_low  picture  "99.999999999" 
a  LOG. 73  say  ")" 
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else 

9  L0C.60  say  lainbda_in_loH  picture  “99.999999999" 
endif 
L0C=L0C+1 

9  LOC.IO  say  "lainbda(tau)  =  " 

9  L0C,28  say  l^unbda_tau  picture  "99.999999999“ 

9  LOG, 43  say  "lambda(mu)  =  " 

9  LOG, 60  say  lajnbda_mu  picture  "99.999999999" 

L0G=L0G+1 

9  LOG,  10  say  "lainbda(tau)  hi  =  " 

C  LOG, 28  say  laaibda_t_hi  picture  "99.999999999" 

8  LOG, 43  say  "lambdaCmu)  hi  =  " 

9  LOG, 60  say  lainbda_m_hi  picture  "99.999999999" 

L0G=L0G+2 

if  LOG  >  pagelength 
LOG  =  1 

if  upper (printvar)  =  "S" 

wait  "Hit  any  key  to  continue  ..." 
clear 
endif 
endif 

♦ - 

♦  Skip  for  either  every  entry  or  for  first  entry  of  month; 
skip 

if  upper(intervar)="M" 

do  while  (month (GAL_D ATE)  =  monthCmdate))  .aoid.  (.not.  eofO) 
skip 
enddo 
endif 

enddo 

* - 


if  upper(printvar)  =  "F" 
set  device  to  screen 
set  printer  to 
?  chr(7) 

elseif  upper(printvar)  =  "P" 
set  device  to  screen 
set  print  on 
eject 

set  print  off 
else 

wait  "Hit  any  key  to  continue 
endif 


Wake  me  up  when  done! 


close  all 
return 

»0^^t0m*********************************^‘************’**********0*’**:**  *********** 
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Appendix  E.  Proposed  Software  Reliability  Database 


This  appendix  contains  a  description  of  the  semantic  data  model,  and  the  representation  of 
a  proposed  software  reliability  database  using  this  model. 

E.l  Semantic  Data  Model 

Korth  identifies  the  entity-relationship  (E-R)  data  model  and  the  semantic  data  model  (SDM) 
as  two  of  the  more  widely  known  object-based  models  [49:6],  The  E-R  data  model  is  very  appro¬ 
priate  as  the  “front  end"  logical  design  that  would  then  be  implemented  by  a  relational  database 
model  for  the  physical  design:  however,  the  E-R  model  requires  users  to  explicitly  define  relation¬ 
ships  between  entities,  even  if  the  relationship  itself  has  no  data  [51:228],  Additionally,  the  abstract 
concept  of  aggregation  must  be  used  by  E-R  models  to  express  relationships  among  relationships, 
being  handled  by  the  concept  of  a  virtual  relation  or  “view"  mechanism  [49:40],  [18:231]. 

SDM  does  not  have  these  limitations,  as  it  permits  the  meaning  of  the  database  to  be  specified 
in  a  more  natural  way  [36:124].  This  moves  one  step  away  from  the  physical  implementation 
description  toward  the  real-world  description  of  the  data.  A  recent  study  evaluated  usability  of  the 
semantic  models  (in  this  case,  the  extended  entity  relationship  model)  versus  that  for  a  relational 
model  by  non-expert  databa.se  users  [7:126,137],  The  results  indicated  that  the  users  performed 
a  conceptual  representation  task  better  with  the  semantic  data  model  than  the  relational  model. 
■Applying  a  'bridge"  approach,  SDM  could  then  be  used  as  the  initial  logical  design  to  capture  as 
much  of  the  meaning  and  representation  of  the  real- world  as  possible.  The  database  designer  would 
then  use  E-R  diagrams  as  an  intermediate  step  to  put  the  data  in  a  format  for  the  physical  design 
[51:228]  Finally,  the  E-R  diagram  would  then  be  used  for  the  internal  schema,  which  would  most 
likely  be  an  implementation  of  the  relational  model. 
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The  following  sections  provide  SDM  descriptions  of  proposed  software  maturity  database 
objects,  Musa  Execution  Time  model  objects,  software  system  effectiveness  objects,  and  logical 
schemas  for  the  different  classes. 
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E.2  Objects  Identified  for  the  Proposed  Software  Maturity  Database 


Mission  Number 


Date  of  Mission 
Tail  Number 
OFP  Suite  Number 


CSCI 

Start  Time 


Shut  Down  Time 
Time  of  Occurrence 


Severity 


SPR  Number 


Problem  Description 


-  Flight  Number 

-  Calendar  Date 

-  Aircraft  Tail  Identification  Number 

-  Unique  number  for  the  suite  of 
Operational  Flight  Programs  on  this  specific 
mission 

-  Computer  Software  Configuration  Item 

-  Aircraft  Staurt  Time  (military  time,  no 
seconds) 

-  Aircraft  Shut  Down  Time  (military  time, 
no  seconds) 

-  Military  time  when  failure  occurred. 

Note:  if  time  is  not  available,  it  Ccin  be 
estimated  using  random  arrival  event 
calculation. 

-  Mission  Impact  due  to  failure: 

1  System  Abort.  Software  or  firmware 
failure  that  results  in  a  system  abort. 

2  System  Degraded,  No  Workeo'ound. 

Software  or  Firmw2ure  failure  that 
severely  degrades  the  system  and  no 
alternative  workaround  exists. 

Note:  Program  restarts  cire  not  an 
acceptable  workaround. 

3  System  Degraded,  With  Workaround. 

Software  or  firmware  failure  that 
severely  degrades  the  system  and  there 
exists  a  workaround  (ie. — system 
rerouting  through  operator  switchology) . 
Note:  Program  restcirts  are  not  an 
acceptable  workaround. 

4  Software  Failure,  System  Not  Degraded. 
Software  or  firmwcore  failure  that  does 
not  severely  degrade  the  system  or  any 
essential  system  function. 

5  Minor  Failure.  All  other  minor  or  non¬ 
functional  failures. 

-  This  number  will  be  different  for  each 
separate  problem/failure,  but  will  be  the 
same  for  each  occurrence  of  the  seune  problem/ 
failure . 

-  Brief  description  of  the  softweure  problem. 


E.3  Objects  Idenitfied  from  Musa  Execution  Time  Software  Rehabihty  Model 


Reliability 
Failure  Intensity 


Execution  Time 

Initial  Failure  Intensity  - 

Present  Failure  Intensity  - 

Failure  Intensity 
Objective 

Expected  Failures 

Expected  Total 
Failures 

Additional  Failures 
to  Failure  Intensity 
Objective 

Additional  Execution 
Time  to  Failure 
Intensity  Objective 

Inherent  Faults 

Fault  Reduction 
Factor 

Fault  Exposure 
Ratio 

Inherent  Faults  per 
Developed  Source 
Instruction 
Developed  Executable 
Source  Instruction 

Executable  Object 
Instructions 

Instruction  Execution 
Rate 

Linear  Execution 
Frequency 


Probability  of  failure  free  operation. 
Failures  per  unit  time;  the  derivative  with 
respect  to  time  of  the  meem.  value  function 
of  failures. 

Processor  time  spent  executing  the  program. 
Initial  value  for  failure  intensity  at  start 
of  operational  assessment. 

Current  value  of  failure  intensity  during 
operational  assessment. 

Desired  value  for  failure  intensity  at  end 
of  operational  assessment. 

Expected  number  of  failures  experienced  by 
time  t. 

Expected  number  of  failures  experienced  in 
infinite  time. 


Increment  of  expected  failures  associated 
with  reaching  failure  intensity  objective. 


Increment  of  execution  time  associated  with 
reaching  failure  intensity  objective. 

A  fault  associated  with  the  original  software 
product  at  completion  of  coding. 

Net  reduction  in  faults  per  failure 
experienced. 

Fraction  of  time  the  program  passage  results 
in  failure. 


Ratio:  inherent  faults  /  source  instructions. 

The  aunount  of  developed  code  measured  in 
executable  source  instructions. 

amount  of  code  measured  in  object 
instructions . 

Speed  at  which  instructions  are  executed 

The  number  of  times  the  program  wold  be 
executed  per  unit  time  if  it  had  no  branches 
or  loops . 


E.4  Objects  Identified  for  Software  System  Effectiveness 


Total  Weapon  Syster 
Effectiveness 


Software  System 
Effectiveness 

Mission  Capable 
Rate 

Total  Operational 
Test  Time 

Total  Failure 
Duration  Time 

Mean  Time  to 
Restore  Software 
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-  Total  measure  of  the  weapon  system’s 
effectiveness  (includes  software  system 
effectiveness)  . 

-  Ratio  of  the  non-failure  operational  time  o 
a  softwzire  system  to  its  total  operational 
time . 

-  Percentage  of  time  the  weapon  system 

-  Total  time  spent  in  operational  test  of  a 
weapon  system. 

-  Total  time  duration  of  the  software  system 
failure. 

-  Average  time  between  occurrence  f  a  software 
failure  auid  when  the  software  system  has  been 
returned  to  an  operational  state. 


E.5  Logical  Schema  for  the  AIRCRAFT  Class 


AIRCRAFT 

description:  all  aircraft  that  participate  in  the  flight  test  of 
a  particular  weapon  system, 
member  attributes: 

Tail_Number 

value  class:  ID.NUMBER 
may  not  be  null 
not  chamgeable 
Mission 

value  class:  MISSION.NUMBER 
may  not  be  null 
Number_Of_ACUs 

value  class:  NUMBER_OF_COMPUTERS 


multivalued  with  size  between  1  said  4 
may  not  be  null 
Flight_Test_Time 

value  class:  TIME.AMQUNT 
may  not  be  null 
Failure_Time 

value  class:  TIHE_DURATION 

match:  Failure_Duration  of  SOFTWARE_FAILURE  on 
MISSION.NUMBER 
may  not  be  null 
class  attributes: 

Total_Flight_Test_Time 

description:  Total  of  all  flight  test  hours  from  all 
missions  for  all  aircraft, 
value  class:  TIME.AMOUNT 

derivation:  Sum  of  Flight _Test_Time  over  members  of  this 
class . 

Total_Failure_Time 

description:  Total  of  all  failure  time  from  all  missions 
for  all  aircraft . 
value  class:  TIME.DURATION 

derivation:  Sum  of  Failure.Time  over  members  of  this 
class. 

Mission_Capable_Rate 

description:  The  percentage  of  time  an  aircraft  is 

capable  of  performing  its  mission;  this 
value  is  user  input  and  not  derived, 
value  class:  PERFORMANCE_RATE 
Sof tware_System_Eff ectiveness 

description:  Percentage  of  time  the  software  system 

operates  correctly  vs.  the  total  attempted 
operational  time, 
value  class:  PERFORMANCE_RATE 

derivation:  (Total_Flight_Test_Time  -  Total_Failure_Time) 

/  Total_Flight_Test_Time 
Total_Weapon_System_Eff ectiveness 

description:  The  effect  of  software  performance  on 
mission  accomplishment, 
value  class:  PERFORHANCE_RATE 

derivation;  (Software_System_Ef fectiveness)  ♦  (Hission_Capable_Rate) 
identifiers : 

Tail_Number 


E.6  Logical  Schema  for-  the  MISSION  Class 


MISSION 

description:  A  single  flight  test  activity  of  at  least  one 
aircraft  for  a  specified  length  of  time, 
member  attributes: 

Number 

value  class:  MISSION.NUMBER 
may  not  be  null 

Date 

value  class:  DATES 
may  not  be  null 
Flown_By 

value  class:  AIRCRAFT 

match:  Tail_Number  of  AIRCRAFT  on  Mission_Number 
may  not  be  null 
QFP_Suite 

value  class:  CSCI_NAHE 
match:  Name  of  CSCI  on  Mission_Number 
may  not  be  null 
Start_Time 

value  class:  TIME_AMOUNT 
may  not  be  null 
Stop  Time 

value  class:  TIME_AMOUNT 
may  not  be  null 
Duration 

value  class:  TIME.AMOUNT 
derivation:  (Stop_Time)  -  (Stctrt_Time) 
identifiers : 

Number 
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E.7  Logical  Schema  for  the  SOFTWARE-FAILURE  Class 


SOFTWARE.FAILURE 

description:  The  inability  of  a  software  system  or  system  component 
to  perform  a  required  function  within  specified  limits, 
member  attributes: 

Mission_Number 

value  class:  MISSIOM.NUMBER 
may  not  be  null 
Time_of  Occurrence 

value  class:  TIHE.AHOUNT 
may  not  be  null 
OFP_Suite 

value  class:  CSCI_NAME 
match:  Neune  of  CSCI  on  Mission.Number 
may  not  be  null 
Failure_Duration 

value  class:  TIME.DURATION 
may  not  be  null 
Sever ity_Level 

value  class:  SEVERITY.NUHBER 
multivalued  with  si2e  between  1  emd  5 
may  not  be  null 
SPR_Kumber 

value  class:  SPR_NUMBERS 
Problem_Description 

value  class:  TEXT 
class  attributes: 

Total_Failures 

description:  The  total  number  of  software  failures, 
value  class:  FAILURE.NUMBERS 
derivation:  Number  of  members  in  this  class. 
Total_Failure_Time 

description:  The  total  time  of  all  software  failures, 
value  class:  TIME_AMOUNT 

derivation:  Sum  of  all  software  Failure.Durations  from 
all  missions  for  all  aircraft. 

identifiers : 

Mission.Number  +  Time_Of .Occurrence  +  OFP  Suite 


E.8  Logical  Schema  for  Iht  CSCI  Class 


CSCI 

description:  Computer  software  configuration  item — a  specific 
software  program. 

member  attributes; 

Ncune 

value  class:  CSCI_NAHE 
may  not  be  null 
Mission_Number 

value  class:  MISSION.NUMBER 
may  not  be  null 
Source_Lines_Of _Code 

value  class:  KSLOC 
may  not  be  null 
Fault_Density 

description:  OMEGA_I 
value  class:  DECIMAL_NUHBER 
may  not  be  null 
Fault_Reduction_Factor 
description;  B 
value  class:  DECIMAL.NUHBER 
may  not  be  null 
Total_Failures_Expected 
description:  NU_0 
value  class;  DECIMAL_NUHBER 

derivation:  (Source_Lines_Of _Code)  *  (Fault_Density)  / 
(Fault_Reduction_Factor) 

Instruct ion_Expansion_Ratio 
description:  Qx 
value  class:  DECIMAL.NUMBER 
may  not  be  null 
Number_of _Object_Instructions 
description;  I 
value  class:  DECIMAL.HUHBER 

derivation:  (Source_Lines_of .Code)  ♦  (Instruction. 
Expans ion. Rat io ) 

Instruct ion. Execution.Rate 
description:  r 
value  class;  DECIMAL.NUMBER 
may  not  be  null 
Linecir.Execution.Frequency 
description:  f 
value  class:  DECIMAL.NUMBER 

derivation  (Instruction. Execution.Rate)  /  (Number. of. 
Obj  ect.Instructions) 

Fault.Exposure.Ratio 
description:  K 
value  class:  DECIMAL.NUMBER 
may  not  be  null 
Initial.Failure. Intensity 
description:  LAMBDA  0 
value  class:  DECIMAL.NUMBER 

derivation:  (Linear. Execution.Frequency)  *  (Fault. 

Exposure. Ratio)  ♦  (Fault.Density)  * 
(Source. Lines. Of .Code) 

identifiers : 

Name 
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E.9  Logical  Schema  for  the  RELIABILITY  Class 


RELIABILITY 

description:  Values  for  measured  reliability  of  softuare. 
member  attributes: 

CSCI_Ncune 

value  class:  CSCI_NAMES 
may  not  be  null 
Tail_Number 

value  class:  ID.NUMBER 
may  not  be  null 
Initial_Failure  Intensity 

value  class:  DECIMAL.KUMBER 

match:  Initial_Failure_Intensity  of  CSCI  on  Neune 
may  not  be  null 
Total_Failures_Expected 

value  class:  DECIMAL.NUMBER 

match:  Total_Failures_Expected  of  CSCI  on  Name 
may  not  be  null 
Failures_Experienced 

value  class:  FAILURE.NUMBERS 

match:  Total_Failures  of  SOFTWARE_FAILURE  on  0FP_Suite 
may  not  be  null 
Flight_Test_Time 

value  class:  TIHE_AMOUNT 

match:  Total_Flight_Test_Time  of  AIRCRAFT  on  Tail_Number 
Failure_Intensity_Obj active 

description:  A  user-defined  value;  future  version  could 
derive  this  from  an  operational  profile 
of  the  failure  intensity, 
value  class:  DECIMAL.NUMBER 
Expect ed_Failures_Experienced 
description;  MU.TAU 
value  class:  DECIMAL.NUMBER 
derivation:  (Total_Failures_Expected)  *(1- 

exp(-((Initial_Failure_Intensity)  / 
(Total_Failures_Expected))  *  (Flight_Test_Time))) 

Present .Failure. Intensity_from_Time 
description:  LAMBDA.TAU 
value  class;  DECIMAL.NUMBER 

derivation:  (Initial .Failure. Intensity)  ♦  exp(-((Initial_ 

Failure.Intensity)  /  (Total. Failures.Expected) ) 

♦  (Flight.Test.Time)) 

Present. Failure. Intensity.from.Failures 

description:  LAHBDA.HU 
value  class:  DECIMAL.NUMBER 

derivation:  (Initial.Failure.Intensity)  *  (1  - 

( (Failures.Experienced)  /  (Total.Failures.Expected))) 
Additional.Failures.to.Failure. Intensity. Objective 
description;  DELTA.MU 
value  class:  DECIMAL.NUMBER 

derivation:  ((Total.Failures.Expected)  /  (Initial.Failure.Intensity)) 

♦  (Present.Failure.Intensity  - 
Failure. Intensity. Objective) 

Additional.Execution.Time.to.Failure.Intensity. Objective 
description:  DELTA.TAU 
value  class:  DECIMAL.NUMBER 

derivation;  ((Total.Failures.Expected)  /  (Initial.Failure.Intensity)) 

♦  ln( (Present.Failure.Intensity)  / 

(Failure.Intens ity.Ob j  ect ive) ) 

Current.Reliability 

description:  R.TAU 

value  class:  DECIMAL.NUMBER 

derivation:  exp(-(Present.Failure.Intensity)  *  (Flight.Test.Time) 
class  attributes: 

Total.Expected.Failures.Experienced 
description:  MU.TAU.Tot 
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value  class:  DECIMAL.NUMBER 

derivation;  Sum  of  all  Expected_Failures_Experienced. 
Total_Present_Failure_Intensity_from_Time 
description:  LAMBDA_TAU_Tot 
value  class;  DECIMAL_HUMBER 

derivation;  Average  of  Present _Failure_Intensities_from_Time . 
Total_Present_Failure_Intensity_from_Failures 
description:  LAHBDA_MU_Tot 
value  class:  DECIMAL_BUMBER 

derivation:  Average  of  Present_Failure_Intensities_from_Failures . 
Total_Additional_Failures_to_Failure_Intensity_Objective 
description:  DELTA_MU_Tot 
value  class;  DECIMAL_NUMBER 

derivation:  Sum  of  all  Additional_Failures_to_Failure_ 
Intensity_Objective . 

Total_Additional_Execution_Time_to_Failure_Intensity_Objective 
description:  DELTA_TAU_Tot 
value  class;  DECIHAL_NUHBER 

derivation;  Sum  of  all  Additional_Execution_Time_to_ 
Failure_Intensity_Objectives . 
Total_Current_Reliability 
description:  R_TAU_Tot 
value  class:  DECIMAL_NUMBER 

derivation:  Multiplicative  specification  of  Current _Reliability 
that  includes  all  members. 

identifiers : 

CSCI_Iame  +  Tail_Number 
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E.IO  Logical  Schema  for  Other  Classes 


CSCI.NAME 

description:  Computer  System  Configuration  Item  name. 

interclass  connection:  subclass  of  STRINGS  where  length  is 
less  them  or  equal  to  8  alphanumeric  cheiracters. 

DATES 

description:  Day  of  the  year. 

interclass  connection:  subclass  of  STRINGS  where  value  is  DDMMMYY; 

DD  is  a  positive  integer  between  1  and  31  inclusive; 

MMM  has  value  ’ JAN' , ’FEB' , 'MAR' . 'APR' , 'MAY' . ' JUN' , 

’ JUL’ . 'AUG' . 'SEP' . 'OCT* , 'NOV' , 'DEC' ; 

YY  is  a  positive  integer  with  format  99. 

DECIMAL.NUMBER 

description:  A  decimal  number  for  eirithmetic  computations, 
interclass  connection:  subclass  of  STRINGS  where  format  is 
number  99 . 99999999 . 

FAILURE_NUMBERS 

description:  The  number  of  software  failures, 
interclass  connection:  subclass  of  STRINGS  where  format  is 
number  9999. 

ID.NUMBER 

description:  Aircraft  tail  identification  number . 

interclass  connection:  subclass  of  STRINGS  where  length  is  less  tham 
or  equal  to  8  alphanumeric  characters. 

KSLOC 

description:  The  number  of  source  lines  of  code  in  thouszmds. 
interclass  connection:  subclass  of  STRINGS  where  format  is 
number  9999. 

MISSION.NUMBER 

description:  Specific  flight/mission  number  identifier, 
interclass  connection:  subclass  of  STRINGS  where  length  is  less  than 
or  equal  to  5  alphanumeric  characters. 

NUMBER_OF_COHPUTERS 

description:  Integer  number  of  computers  on-board  the  aircraft, 
interclass  connection:  subclass  of  STRINGS  where  positive  integer 
is  between  1  ^Lnd  4  inclusive. 

PERFORMANCE.RATE 

description:  A  ratio  of  time  values:  non-failure  operational  time 
per  total  operational  time. 

interclass  connection:  subclass  of  STRINGS  where  format  is 
number  9.99. 

SEVERITY_NUMBER 

description:  Integer  number  of  severity  of  the  softw2ure  failure, 
interclass  connection:  subclass  of  STRINGS  where  positive  integer 
is  between  1  and  5  inclusive. 

SPR.NUMBER 

description:  Software  Problem  Report  Number. 

interclass  connection:  subclass  of  STRINGS  where  length  is  less  than 
or  equal  to  8  alphanumeric  characters. 

TEXT 

description:  Narrative  text  describing  the  software  failure, 
interclass  connection:  subclass  of  STRINGS  where  length  is  less  than 
or  equal  to  80  alphanumeric  characters . 

TIME.AMOUNT 

description:  An  amount  of  time  expressed  in  hours  euid  minutes, 
interclass  connection:  subclass  of  STRINGS  where  value  is  HH:MM; 

HH  is  a  positive  integer  between  0  and  23  inclusive; 

MM  is  a  positive  integer  between  0  and  S9  inclusive. 

TIME.DURATION 

description:  An  amount  of  time  expressed  in  minutes, 
interclass  connection:  subclass  of  STRINGS  where  format  is 
number  99.9. 
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